{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PMNToRD8EZFk"
   },
   "source": [
    "# Comparing ModernBERT with series of Bert Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "poP7CJrfFtZj"
   },
   "source": [
    "This notebook compares ModernBert with several established BERT-based models, such as Google's BERT, ALBERT, and RoBERTa, using the [Uber10K dataset](https://llamahub.ai/l/llama_datasets/Uber%2010K%20Dataset%202021), which contains question-answer pairs. OpenAI embeddings are used to represent the questions and answers, allowing us to assess the performance of ModernBert embeddings against OpenAI’s. The comparison focuses on key tasks like question answering and semantic understanding, evaluating accuracy, efficiency, and overall performance.\n",
    "\n",
    "**The goal is to highlight the strengths and weaknesses of ModernBert in comparison to these well-known models.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mUr2MCirFhzz"
   },
   "source": [
    "![modernbertembed.jpg]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ix4x_JFnFj7Y"
   },
   "source": [
    "ModernBERT Embed is an embedding model trained from ModernBERT-base, bringing the new advances of ModernBERT to embeddings!\n",
    "\n",
    "Trained on the Nomic Embed weakly-supervised and supervised datasets, modernbert-embed also supports Matryoshka Representation Learning dimensions of 256, reducing memory by 3x with minimal performance loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5zLwbVtNHGwm"
   },
   "source": [
    "## Installations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mqNtX_LwHNE0"
   },
   "source": [
    "We'll use Huggingface transformers for extracting embeddings for all the models. For this we'll install latest version fo transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "15dscb9Y063z",
    "outputId": "8334c03a-e373-4703-8b58-f8e66a245a6e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/huggingface/transformers\n",
      "  Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-ie06m4wt\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-ie06m4wt\n",
      "  Resolved https://github.com/huggingface/transformers to commit 33cb1f7b615f59f3d18de1f13bddfcbfa6b863a5\n",
      "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
      "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (3.17.0)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (0.27.1)\n",
      "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (1.26.4)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (24.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (6.0.2)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (2024.11.6)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (2.32.3)\n",
      "Requirement already satisfied: tokenizers<0.22,>=0.21 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (0.21.0)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (0.5.2)\n",
      "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.11/dist-packages (from transformers==4.49.0.dev0) (4.67.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.11/dist-packages (from huggingface-hub<1.0,>=0.24.0->transformers==4.49.0.dev0) (2024.10.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.11/dist-packages (from huggingface-hub<1.0,>=0.24.0->transformers==4.49.0.dev0) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests->transformers==4.49.0.dev0) (3.4.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests->transformers==4.49.0.dev0) (3.10)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests->transformers==4.49.0.dev0) (2.3.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests->transformers==4.49.0.dev0) (2024.12.14)\n",
      "Building wheels for collected packages: transformers\n",
      "  Building wheel for transformers (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for transformers: filename=transformers-4.49.0.dev0-py3-none-any.whl size=10582079 sha256=bfb08d3dc51c5627b7065a3a328e793a6ee5a491f39ff6fc7726112a4525e9a3\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-xuu9ewv_/wheels/04/a3/f1/b88775f8e1665827525b19ac7590250f1038d947067beba9fb\n",
      "Successfully built transformers\n",
      "Installing collected packages: transformers\n",
      "  Attempting uninstall: transformers\n",
      "    Found existing installation: transformers 4.47.1\n",
      "    Uninstalling transformers-4.47.1:\n",
      "      Successfully uninstalled transformers-4.47.1\n",
      "Successfully installed transformers-4.49.0.dev0\n"
     ]
    }
   ],
   "source": [
    "# install latest transformers\n",
    "!pip install git+https://github.com/huggingface/transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l5LqPknvHluN"
   },
   "source": [
    "Install all the dependencies related to llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "H1sg6hzaAkty",
    "outputId": "69be1ee6-a2c3-4a3e-8886-5671a4dbb0a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m32.2/32.2 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.3/38.3 MB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m43.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m298.7/298.7 kB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m22.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m58.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m248.0/248.0 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.8/50.8 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "# install llama-index and related dependencies\n",
    "!pip install lancedb llama-index llama-index-core llama-index-embeddings-huggingface llama-index-embeddings-openai llama-index-readers-file llama-index-readers-llama-parse llama-index-vector-stores-lancedb -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S0vyfPE1HtgD"
   },
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "up0MxfPJHv47"
   },
   "source": [
    "For this comparison, we'll use [Uber 2021 10K Dataset](https://llamahub.ai/l/llama_datasets/Uber%2010K%20Dataset%202021)\n",
    "\n",
    "\n",
    "**Description**: A labelled RAG dataset based on the Uber 2021 10K document, consisting of queries, reference answers, and reference contexts.\n",
    "\n",
    "**Number Of Examples**: 822"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aNnndaLRPQLp",
    "outputId": "887da6a9-3e24-42b5-a702-58a20123d152"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100% 1/1 [00:00<00:00,  1.66it/s]\n",
      "Successfully downloaded Uber10KDataset2021 to ./data\n"
     ]
    }
   ],
   "source": [
    "!pip install llama-index-cli -q\n",
    "!llamaindex-cli download-llamadataset Uber10KDataset2021 --download-dir ./data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RERDlsoYIgay"
   },
   "source": [
    "## Comparitive Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KnKwti49CvqJ"
   },
   "outputs": [],
   "source": [
    "# import dependencies\n",
    "import time\n",
    "from tqdm import tqdm\n",
    "import re\n",
    "import unicodedata\n",
    "import pandas as pd\n",
    "\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "from llama_index.core.llama_dataset import LabelledRagDataset\n",
    "from llama_index.vector_stores.lancedb import LanceDBVectorStore\n",
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.core import SimpleDirectoryReader, StorageContext\n",
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "from llama_index.core.settings import Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p32Nw5WuCQNh"
   },
   "outputs": [],
   "source": [
    "# helper functions for evaluation\n",
    "\n",
    "\n",
    "def normalize_text(text):\n",
    "    \"\"\"\n",
    "    Normalize text by converting to lowercase, removing non-alphanumeric characters,\n",
    "    and removing extra whitespace.\n",
    "    \"\"\"\n",
    "    text = text.lower()\n",
    "    text = re.sub(r\"[^a-zA-Z0-9 ]\", \"\", text)\n",
    "    text = unicodedata.normalize(\"NFKD\", text).encode(\"ASCII\", \"ignore\").decode(\"ASCII\")\n",
    "    text = re.sub(r\"\\s+\", \" \", text).strip()\n",
    "\n",
    "    return text\n",
    "\n",
    "\n",
    "def evaluate(\n",
    "    docs,\n",
    "    dataset,\n",
    "    embed_model=None,\n",
    "    top_k=5,\n",
    "    verbose=False,\n",
    "):\n",
    "    \"\"\"\n",
    "    Evaluate a given embedding model on a given dataset.\n",
    "    \"\"\"\n",
    "    vector_store = LanceDBVectorStore(uri=f\"/tmp/lancedb_modernbert-{time.time()}\")\n",
    "    storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
    "    service_context = Settings._embed_model = embed_model\n",
    "    index = VectorStoreIndex.from_documents(\n",
    "        docs,\n",
    "        service_context=service_context,\n",
    "        show_progress=True,\n",
    "        storage_context=storage_context,\n",
    "    )\n",
    "    tbl = vector_store._connection.open_table(vector_store._table_name)\n",
    "    tbl.create_fts_index(\"text\", replace=True)\n",
    "\n",
    "    eval_results = []\n",
    "    ds = dataset.to_pandas()\n",
    "    for idx in range(len(ds)):\n",
    "        query = ds[\"query\"][idx]\n",
    "        reference_context = ds[\"reference_contexts\"][idx]\n",
    "        query_vector = embed_model.get_query_embedding(query)\n",
    "        try:\n",
    "            rs = tbl.search(query_vector).limit(top_k).to_pandas()\n",
    "        except Exception as e:\n",
    "            print(f\"Error with query: {idx} {e}\")\n",
    "            continue\n",
    "\n",
    "        retrieved_texts = rs[\"text\"].tolist()[:top_k]\n",
    "        expected_text = reference_context[0]\n",
    "\n",
    "        # normalized retrieved texts\n",
    "        retrieved_texts = [normalize_text(text) for text in retrieved_texts]\n",
    "        expected_text = normalize_text(expected_text)\n",
    "\n",
    "        is_hit = expected_text in retrieved_texts  # assume 1 relevant doc\n",
    "        eval_result = {\n",
    "            \"is_hit\": is_hit,\n",
    "            \"retrieved\": retrieved_texts,\n",
    "            \"expected\": expected_text,\n",
    "            \"query\": query,\n",
    "        }\n",
    "        eval_results.append(eval_result)\n",
    "        # print(\"Retrieved: \", retrieved_texts)\n",
    "        # print(\"Expected: \", expected_text)\n",
    "\n",
    "    return eval_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "5670bcbf00b24022b17567fd10547c89",
      "c2ed3ccb928b44a38f38504e4963274c",
      "24b5d037519b4b1bba57bd08db189cef",
      "835293d111a4429697c74edb4179b81c",
      "18e5e27d0de54fb3bb5926a2f6c2f733",
      "16a2cde17b014ceeb75295777a6cd527",
      "812ca379f2ee402db5267f3c48af013b",
      "4aeb7c89285342ceb69723751439ccad",
      "5dd45d472d3c45faaf7fc4a3a290cfb8",
      "546a3b9f661c4a39b47746cba3ab7990",
      "c7e8f8f235bf42cc86b4fa02c170335c",
      "f70b425fbc2b4aacad9354b4c70fe79a",
      "e97e9e409317469db586cf8525a983aa",
      "a8f6a165f71346d69e94ae279996800c",
      "ef1b1e7fe37048a09f7e6ae0cb133e12",
      "27d505e4b0554c94987c7087d5c9050d",
      "fba5ddb1b0a245908d2325bb092fddd2",
      "a2d4a7b4b52c4cb2b354ac177ce898f2",
      "05bed16034e64e688625ddb023874311",
      "b3c42a8722c04ecab187d12f509bcdda",
      "9b91adc6c77c47a6ba6fa3556261084b",
      "67e334ae3c7949de858f666b8041544f",
      "336b87b35f8c48aea2618577f33ab0e1",
      "e2c3e7eac50f44c08b85f844bbb3a9be",
      "2a63fe21a00a40a6b53a9c15cfee209d",
      "3002ed9b87174f42a081a212a3773464",
      "a197423d07d7495ea41263cf51eb581c",
      "a87a92e955024284aff56adb4d8ffb7b",
      "cdfc31ea89d54e59b19404065b851631",
      "685fc13cdeb04fdf8c8f91c9c66796d4",
      "8ff54b61d7cf4a6bae20831b2c87d1b0",
      "91074ef330204f858d97aaaf6c650c77",
      "56dcc737b0a14cd4915bc78dcc5e8175",
      "f7a86d8a074b4c649bff073e6532e6a5",
      "5afe074b545d4ccebaf374cdd366c355",
      "3d11724e212445ec8c00d6848948915d",
      "17518afc12f64580bd81e273b6881bd2",
      "0fa2a93df2a641d7bd108b95be28d44b",
      "9faa5edd98824311be168a0c823d1706",
      "63b7695484d14772a6f3636d7eb02bd2",
      "31c2bc07773e4d5aa8f11aeaa5c7124c",
      "83e81993ad104630afe443030238e01a",
      "7a94bbcaece24c668603c24ff2ed81ce",
      "bd24c843e81e47ad85af72d4412496bc",
      "381e0656223c424ba96daa2c43b92a59",
      "91c77badbbe04dd19689d65517a231b5",
      "c73964790e2a45258c07d79ee9b349d9",
      "99537bed53304c3abd109cb9c69971bd",
      "763b0e678aee4b4a97f590d2b18dd88e",
      "f69c9cf154fc4f9c847277b5d6f9ca09",
      "88ac310eae224fa4a7c27979a593c60f",
      "2adae71d2e6b4844bfee0cba3d9b1746",
      "da84c2cfd7e5473f9ca413c4662094d3",
      "64ad457d4b254986bbed2e7ba8960574",
      "be6f5fb765bd4651b7f34131be8d1f04",
      "143911292d4b4e82b7e47b06fd94fc03",
      "f64db66a19d343fe9ba778232cb71ab0",
      "76bf0f2b4a974bd482c09245755f92e3",
      "a9197ce8a96943a1a4767d03a776189d",
      "4a6a9908a0444fff95c8a11d30df1c62",
      "08e25429ad3846b9a747d38fe4739e55",
      "5a8ee9142b99448d9fedb3ee7ce7d395",
      "6cd1b0a0a266410abd534af6efc87ea5",
      "7cd5e411780a4dc287a42daf31351f1d",
      "050e5ffde6614aaa9bfd1230ccb15c66",
      "6bc4938e09f54f61a7f23056b0b385e4",
      "0b8c1819547244b59e8db6126053499a",
      "81770c016fe94a17912ea8559b6d6ea7",
      "ea972db8c17e42c9846273ac23362431",
      "b9687c3244fb46c9b0b3e915adb06863",
      "569ab09c946c41fcbbe6997fbb202782",
      "01dae09d89884ddcb2c419ea5c9599d7",
      "c27d1664d8f145c89c0676d1aa0d1d05",
      "dbb329eaf50a4e2c8f2f6b7c3896c152",
      "a4c55c24c88043439e792accb3b63219",
      "5f872508e06543ccb867c67ef5374f9f",
      "ac6e0d67cf604d6ebacfe5a1fc10c27c",
      "a8bdca71c87c4d60b8f8bd0087503ff8",
      "e7e9f5c606ea4452a7f4eeb2f41758cb",
      "88f8be740fd34490a449436f887d33b6",
      "a93d1e749c114f30a1449ce7b3f5bcf3",
      "ad62d454ce4b4e289ad87abed3122e52",
      "41387d280e61419d9ffac7f172b97bd6",
      "ecb6dce60e4b4d48b085a4573e187ae2",
      "b8ec43b0c31a406b8faab5d846f47f0d",
      "b24dfffdf6724bef99ac28139bbef237",
      "1aab98842b204c18826b2655d60ac806",
      "fe7d7b597c914dd1ada6fafef4fac994",
      "c5dea460c2b94d959d6f667de1a48773",
      "e0198bb7de914c03a3bebb1ee4935e8e",
      "e9c24d66009b4f5abbf2f5f57b72f7c5",
      "3a70a0de1dcd4de5ae5a90aa40466f89",
      "a6d0eaddddbc4eb589476ea6aadfe092",
      "47cac3bf96574e49a05502d57786df21",
      "6d3841d7eee94a728a354facb7c16019",
      "319315fe30994eb5a7ad464821f2b07e",
      "be32874e95f1494faa814130c044278b",
      "52e2785c4aca41d1bb667420f70739b8",
      "923a92c467134a2aa6b1fd9d329ac46b",
      "087d7bcba0ba4f2f8034087a29cd8242",
      "aa7a03bc624043bf8e368341d285bb2b",
      "ead1c98a8a6f4ee9b04c4a6ca6cb9d0e",
      "ad6d666ee97b4af9a7e970b31e37fe34",
      "3b0595143b4647c89e57d8265014dbb6",
      "97b075fb375a4c889d3981e0e1641c07",
      "20755fba4195419d9855c8ff0151beb4",
      "b3ce9d10a51e4704a49778d08dcf32b6",
      "44281930179349c99bd4ccb21b8b2891",
      "066b3652b892423680cbc3a57dd84dd9",
      "86ee448466c94a2c84eae56ad15e446e",
      "6ec58c1fdcf04fd48d6cb22948f7048a",
      "497001846f244f999d5d310117a825aa",
      "e6359e1013754aeebfdac5fb4c81cfd7",
      "c1b1d8151869468fa4a5fb1265fc1388",
      "bf32908cdc24428e97d21e63fe62ab15",
      "e777a7406e5d495ea1920cc4a182d920",
      "a39067ccb76040139012b9be39be9c57",
      "425a536eec4341019262eadc279617b2",
      "9da61078a319477db6db00f7432f8586",
      "8ebfce0c36284a64a870f7006e8f2e81",
      "315b2205e5214073b4522ca8ffa9d9c1",
      "a14b1821949d45da90d4f5955e4c7ebf",
      "4cbd7f8e48294b5ea00d95706eba3e08",
      "036e607ba24b445a927592404293c04f",
      "a7a896e9b3064d519b634410d76211f0",
      "621044d5b9c14ba997a10ce9c6146064",
      "94cb8c8374cd4c11943b1baa656b3e91",
      "91e289680d984c9692048711527c1822",
      "3c606e008f484b0c9e24cf60ab6ea04f",
      "99f4be25b0b7498088f3bec7029ef083",
      "71e972db67cf48b490c025ccf9a937a9",
      "8b7cac0500ed4e528d07c82eb03822bf",
      "77318d9f00e443e18708ce9a1457f872",
      "0a1d28453b3f481b823f90cb063fa6df",
      "9a9ed980596240b3bf2f888795ccef08",
      "42c0228608424e7898eb41ab4bdc2fef",
      "9ff5765fe3434599a0ba9f266468f2b9",
      "50a10a0fc10f46fdad7e31a443a586f4",
      "0bed8eea3590458dbd43ed8b249e2723",
      "0509a8d4c388412eb6da27455bd8a89f",
      "925c55ebe6e144869bafe0bcd2902ef0",
      "0650e0a3e13f4614833c04863968318b",
      "49a4a57ec6e347f785412d5e99bc61f5",
      "bbad62e82ccf4a46aa7d7b717a04f3c6",
      "d9694b08b5ef483f9089ad3c7fedee94",
      "d8cdbe8266654cd49f2de34554332bef",
      "479304b685474f60b42fd074ab20820a",
      "0a9cb5bb624b4db1964274ae56f86649",
      "582f6483caaa401394557fe91c31a289",
      "5a2151702a4a4e2ab5245e177ed0be29",
      "eb57f25f4b9949768de982c4bece5c3a",
      "9a0efa5863dd4861ba240b43616a6e20",
      "9097baf51799440593f97db5746ed8e1",
      "036838d4e2c1454281ae5d29fd03ef1f",
      "afefadd644ff47ccad645acdae3cd42b",
      "6ce13ecc9ea944ef8357c118b2e42387",
      "d02bd53690884389a42f33eedaf68bb0",
      "3b204cd6077347f0a902529f47f78a79",
      "cd5c0f8225004a20a37282810ada2d0d",
      "bb71cd9074b64ee9891974b336aa1da0",
      "38f4155e3bae4ca3b2dee9d1e434e84f",
      "c321d47155e2434c9d87d2fc1a1bcd3c",
      "19ded2e88b184f20bf7e43a2ddf88e00",
      "6c9f9d9b89344ba5b1dc9bdc387992c2",
      "bfd6c3961f2d465eaa28f6bc54782f52",
      "bcfc0cdb7c994b2aa639a1170e37dec9",
      "4281fb7644a74b3680673c992ae480f4",
      "03fc518a615b4e29b2b1519bcdc35639",
      "348da7cad4834754adab5c7127555a22",
      "b7b4f7e59b6d4c85bf955d94f349a433",
      "50f4e687c37b42e782c5df293e157bcc",
      "e4ee0306599d4b3ab5c2bc11b84b5df8",
      "ccade6725cbb437f915c0f51e746c06a",
      "f0464fb44d6447d6a6fec35c8615742a",
      "259839582a424fdaae627a0ac9db33f4",
      "3c27665e671a4cf2990403f50f28f4f1",
      "e20b0f46922f42c2b76359cf39380595",
      "96037ff91b224d108f1cbd4b94e566f3",
      "01c4f50eef3449fb96687d82a057baf4",
      "2446979ec401407b9e34783d1b4a91c5",
      "41770f8a7f8b4004b3354420713426d4",
      "cc671057897d4860b002cf40f1755c4a",
      "7e143c6381964501aa50fd6c1472a881",
      "d7a2f85070aa4deb808c7f29b47a8ac1",
      "405678cd02ea42b0881636ec5008f424",
      "a72ac267f62b456bafd6f28e11cc36ab",
      "0c625f924e744671b4499e8f3019b57a",
      "5d86ded4a1b747f1a7609606d9368e4f",
      "df6c5337c6f540fb8e72cef44498b346",
      "fb5f4901dfa34953b8f38c7adf69430d",
      "c49dbc551c004236a80faf7e03a5f932",
      "ed343020dcbf40aa9bcd9b8db9d3013a",
      "5449cd3403964962aaec59f8d94f03b3",
      "65d056a38e424c67a72851ae3da5017e",
      "1da5a2ae82ac4067acd5a17fc0528bc3",
      "46c9131540d64a92b6849f17b0a412de",
      "19bd63eeede447eaaad21b28e2655c09",
      "266625fbf9344fa68ec11dbee0dddb26",
      "847de608f72c42fab3c954b0aaa79e33",
      "47ed3f79152d43f481fd63efa47d9b7f",
      "41449fddfe354aabbccd1c8cf8be71de",
      "301dfcca1a974d308f2a5aa86c2349f5",
      "ec690b64a73f48e089701638b88907ac",
      "6467b372ea244a2c91ff1db482766125",
      "01c1005a042e4ea28529f383d300a1c3",
      "6249f2f917174ade8da78540caf24052",
      "0cb7bd9d81ee48f3913f42da423309a8",
      "f5e6e6644fb44be1ac3c19c6e9953db8",
      "90539d882a7e44599154a2f995f942b3",
      "0a4920ea1c1f4b86953467bfcdd95b82",
      "268171a340424e008ffceaa17827d44a",
      "f37977cddbb5469790f04cfa3962e25e",
      "108fbe4a972746cfb8796fd303dd17f7",
      "8bf597c800e348da8fe692fa0edde613",
      "7f50e88caae8443faf6a8b995d01c41b",
      "8b31bbe7f49b4bd88927472ff5b6a6a4",
      "156a2a9618ba4febba4d53159848b0fb",
      "935592bec4bb4a58bd47605599c838da",
      "cd5ed96bf0d245c1bde0bd39068f6412",
      "3af12298d09449d08ef5b0fd5a85db61",
      "9d550bebe5a84b12853b6ea916da1db3",
      "0cb1ef63f233404c8b17911a255ec5e4",
      "d37cc0f032d54b92b9322babd812fa7d",
      "c24100fe2706488a86bc21f498701d04",
      "9e84b1db92bf4cb8872af2b1af160df1",
      "1d4979bba79f45fbb7f4937a02d1fa5a",
      "0312f7520ec94abd8cf38c33cee7ba88",
      "22d4dcb46fac4825b281208de3d29c04",
      "aa76a95232524429b566e76f0eec617d",
      "8dcc7869c38f426a932d0c74cbe3aa62",
      "cf898578fce3456d9c1894e432538cc6",
      "994e1382f9744aa19b1d60c50497bdf0",
      "a3961ee29b334afea20c590f3e4e7efa",
      "8a03de742fb745e1b41e36c5c80c804f",
      "93cd943ea6ae4d0bb0681b722cee92a6",
      "e753b405f42949169868bd6aa61fd2bd",
      "ab1b0645628d42d9ad181d7e301a21ce",
      "e26944e6e67d4317a8b9644e9aae3feb",
      "1df48325f4c148fcb599b9d1a0be11c2",
      "225a54f1faab47a88acdeb0696232f0a",
      "4a91044614c4491d8647b1ecd87393ff",
      "d5fea1e5890f41afa92eb6f0b577b507",
      "47a0b16fee4a4eb3a187250a875d0f7c",
      "40b7e5238e9e4b59958643cd84a6b649",
      "d40ca2336fa247039327f270541054a6",
      "e93e87739f5a483fa3820873a06950c1",
      "09639ffc58d0495c9fc660226009cda6",
      "35b44aeb78614d3f87827bd3e6d2b48b",
      "bb69bed37c8e4a08b906a8b1562b158f",
      "754c8a4e953141bfa46d8061fe944d9c",
      "ad3fe4ba40794d83815cb8afcb6eb060",
      "22c81beaadf647af8da7b389ee04a5c6",
      "b2a92abd82e045c9a34ce477b9b36b53",
      "43952711f55743c392025a14b01f9918",
      "6d83170c399646f882c35eba9d90d883",
      "37c4416d6d4d41789202f664e4527f0e",
      "a92aeb8e64c24310a57030d659cd8ee9",
      "dcd46190180e4cd5ab3fa755959220b1",
      "64673909529d45699b734e52db0f3947",
      "01900920c7434dc892e38316bf4510c1",
      "b33fa37cc6094bff95dbeef9aea85cf5",
      "d4d9e2bca65c419bbeaecfd803e4f20a",
      "596192ddc64341d09756fa1a47f8f35a",
      "4c99d6e3539246deb473623e8de3e513",
      "a221d52896234533a6e43edae2efdb70",
      "be312e2d0fc64ab7bdc551ec8032c971",
      "b14593a699234311aa9c8c1b9909a585",
      "8ca7d3ced2144740a6e6b9ecb365fceb",
      "24205547ada14faf921e242eaa55db57",
      "a6141cfae3b6456d8b7df906999e4d67",
      "5ccfad2d8bfd48b186491358945c0d09",
      "f8cbfd1e8c5a42f8a356e8e215b220c5",
      "7894d08b4db94b26b439a7c83eb9ac81",
      "9c73ba310db8403398fe46c9512062fb",
      "12db06fa9fd14575a954ebf336f35980",
      "12ababec540b41b39be575d66cdb7ea4",
      "1452bd906f104b4b904f316c44ae3da1",
      "ede695e46be54d15ad5ce0ac7089ea69",
      "2588418943ba4eb8b5070b00f2dabc71",
      "50d3ec4660be4c2ca4712a39c80312cc",
      "f8c0338b3c66496ebb83d263a4f748cd",
      "b26de9763eab4619929273d03939a15b",
      "d543fa5c88e6488eb458a22965fc85d4",
      "5e668a238eeb4234850e6e8aced0373a",
      "e515144ae3264e97800f1480cfdc05d9",
      "0b4f6eb6a507435da3a43d04d980f015",
      "5af7ba52959a49ec9d6ce318ddcc7481",
      "592be1f059ac48768714473f1db6f682",
      "6f2e8ef63a72443b8a6201dcb5d2330d",
      "17efc0c272344ed28e1bab0d0b09a104",
      "ce0c8befd2c74280b7d8ff32fb6279d9",
      "7862e32f38724c339ac97455dc2ae04c",
      "5d9fff9e7c72490699725b7188297186",
      "973358514c234f17ac3b3141904afec8",
      "08aa75e310f34c69b2c7849733cf829a",
      "0c2490ef3bfe44beb01bd39fd5c39139",
      "92078f01b8164ba28bf4f088b8a71d39",
      "df7c9f909d16407393282a17af0764a1",
      "535cf91283e94bb8818ab520cc75e1e0",
      "482a409f1dbc4537ba8b887ee5d11019",
      "ed94200e435c428a8867b8955c2b6988",
      "833d8efdfa05401fa4316c3d7156f305",
      "784206d4082b45f0b3e3adc6d70ff234",
      "2783928323724c2da8c44f7aaa0b59b2",
      "36fe46c4b5504f87b1ea45149f6f8d67",
      "ac1a95c4708448bbb1510827741240d7",
      "18ce7ea1d13b42229d49dbce7793d9bf",
      "5deecdd7dbd840dbb1eee468e34040c6",
      "9376a37a628f49aaaad8da3e43eb80f7",
      "23a6ae9640d6420d9098bcccda9bdf27",
      "357259bda32041c0a063c053d1ca8e12",
      "62aab10c42594a1d8600b61fe60d83d6",
      "6d2ff5c5847e438dbc4e8528e81c8990",
      "f7893f93880844c29cb07950db423b05",
      "23be0c68ce80453097e5909d588da952",
      "7fd26e67b7a444e3aaca97dd2ef8b852",
      "d190377dbcdc4e7ca0f795058346efab",
      "b7e5954391014e2ca17f6342ab1eca5e",
      "0283f948c7904ee89a7f8275c3769d7e",
      "13ea642d38994d3880b49551f1d19a2d",
      "d1f093f9f0dd4463b4d457afefe73ae7",
      "e942975382a340448eba06ee698964ec",
      "959dc8f030e74404991bcc57a84df4f7",
      "7e9b0ad967934fa6849b89a655bc5abb",
      "4b744ea18d0c4b9daccfd19217fbbbb5",
      "7cda6740e126463ba59f62161e3c0f3c",
      "3b033084f77a4b419d0798f8db07af57",
      "e1a08c4082b84d59befaf6a33bda2808",
      "a5d1d8317345451b8e8391ca86c89c2e",
      "2cc293f65a554a6186fd5b4050a9cda7",
      "3051bd88387d469fb6cafec7f65b5913",
      "02dd423d6f6b4cc1b5a074e840a22ede",
      "03e1904b40184a6da96f93ea3ce6ccb8",
      "20f3f2f85a804579a97dc760e0a0500d",
      "e7177694eaf94d039d8f91091bfae201",
      "0357db5d4a944aaf8b575d83d8e25023",
      "9112b56cc61e4cd7aa197198034c1065",
      "e4c662443a1241e181e62cfa6d9e729d",
      "43e43da283ab4cd6badeccc954672247",
      "b6478549b1ee4dc9b760e643b0f04d03",
      "360f8000c8d84c4b86ee3c65244519cc",
      "b62c95a898c74effb14cdc4603d4b9ef",
      "7ffde501cbe645d2a65f0ad2e057ed6b",
      "4c8acaac00aa4627b783132e8f4213f6",
      "d2249949b6f245c7b1a17ce69338cd01",
      "7dcc090c580741408f027d55d7517c1e",
      "9e22b8e36c6741008fe310a84805c0e2",
      "deba32d2b99a49e492acc61f153fcb7f",
      "e5e8e77dc8b442129cb5377a788b33dc",
      "c53454928ae14c71a299c449e5ef4fcd",
      "b4e94c8c76884aa59e0c7d37749b5ae8",
      "fd535091190a468daf4389c0853978c9",
      "025fde342e3044789e622405b22978de",
      "bb0975d7ed1d47c4bec517fcd082b8b2",
      "a915e70ba9cc4ff1903e400a0af5af6a",
      "e6033ae606c04421a148d5bb7bb91e35",
      "ed78e4e7ec1642e3b7971a167a55eb4b",
      "9c895d1cf2ab47b3a64c6950e0471c73",
      "46b68e35acba4569a664396e542d9cc2",
      "ab0ac4e17a7749b49131f4ecd7faaa61",
      "b40f8e7dc0d24fc893bc19ddd00e8283",
      "6eefee0e215440a2bf3f02e1e9bea63d",
      "6d4f79c8319e4bcea988df3b17f8110a",
      "39d98c0c95e34508a65cf475ef86a9db",
      "9d2a7148be704c50b10ca7b3e7031d1f",
      "756eb40947f841f6bbfdc0380d58edee",
      "8556e6f5fe5c437082f95a1d2d5c4ad4",
      "0d138ed3dfcb41f786d3bc96d82bed3a",
      "85534864a37a40adaae6658f8ee0cbe0",
      "5e4c7508e2fa45da9aaf6ecdec1ebe38",
      "428eced7a5454e70ae4d6bc5b1de8515",
      "e1270dfa3e2342e19e0afd43161c9d88",
      "3bcf0b6d71714d52af0b266f109f8174",
      "688eee1cc3e94fca808415b82a16c41f",
      "f8f8cee3d4be4c1b9d3b0218f2fb1f93",
      "7add9a693a964c6ea9743320ea67dfcf",
      "bb89b702268540999054e7b61d814784",
      "c35c168f8c0e477c924f1dffe52025e3",
      "2d5e47ab3e474639a0ebf3b8ccb78338",
      "0b46f166b06a4158b4402f45d8454313",
      "c593fd559a4f4e06b7209fb408fe8ff2",
      "139a9fb013084a8380a55f5accd56666",
      "573ce36aebd44265877c445350b1782d",
      "e8347c4a221346f09a23645fda5e1702",
      "d541af02ff6a48aa80b6450476306c6c",
      "ba54efe6c20b44bb9b7e68af4f445e39",
      "8a736a1f97984cb5b1b2352727f95ba1",
      "564a8c7144a94ab09eadf3ce8b064e27",
      "c4eb279236324ab28d072b61c92d82e2",
      "2721b5e373d54b97bd2936884293df2b",
      "501c3be4fc9e43e0a4b8662a1791aa6c",
      "e040aecac6e44c198994c77346779ba4",
      "60b7ee0eb1a74709bd94bb422e0e367d",
      "a7c6602715f04eb896e84ac837b37f11",
      "2f4d20f610fc4915850dcf0c801b2d65",
      "b5e6b2a154744ad190c0e3e7a928f8df",
      "3bbe9bb826dc42cb895e1eff21540f66",
      "4601c927c25341c096cdb18592cfebc1",
      "8493d29e79cf471fb6cf6171b5f44056",
      "252af78b6ee9459eb8a3d5fb0db31a03",
      "9c095f1acccd49ac880f79ffc6717d96",
      "859d274746ca451c844d15ba4cf07946",
      "f1ac7b10de114ad6b90cf01722d749c8",
      "0b4718df0d524df185273b3901eaa431",
      "cec2d3b11e764818ac2b0ee278146481",
      "8862728654bb460ba077d0da8ac877f3",
      "e112e9249869449d95fff0c4f8f4d3c9",
      "1bca72e65e3340369e3e9f61a7781f4e",
      "236a1e42db2a468ca4bb669653fa1389",
      "d2d55122c81e487bb85503213f1f8131",
      "bcd3c453cd4d416f914bb725269ab9e5",
      "a23618e620b54fe088107d031a1f4623",
      "a8e84d1c7deb4c279e9cd68ebfa12eb8",
      "33af45a085c443b0a03687e0a75b0ecf",
      "f0f3c0cdf72045b5929811052c7baaa0",
      "127f8941e48c4226b2e6be59e16c1bd6",
      "fc0c984ce44c44cab676ef087f7fa157",
      "26630bd4333e4bc28df0056f25f25425"
     ]
    },
    "id": "pBpOmHwvR4ox",
    "outputId": "87b16588-fb56-4186-8f80-f04ac011a613"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5670bcbf00b24022b17567fd10547c89",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f70b425fbc2b4aacad9354b4c70fe79a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config_sentence_transformers.json:   0%|          | 0.00/210 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "336b87b35f8c48aea2618577f33ab0e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/445k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7a86d8a074b4c649bff073e6532e6a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentence_bert_config.json:   0%|          | 0.00/54.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "381e0656223c424ba96daa2c43b92a59",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/1.26k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "143911292d4b4e82b7e47b06fd94fc03",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/596M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b8c1819547244b59e8db6126053499a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/20.8k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a8bdca71c87c4d60b8f8bd0087503ff8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/3.58M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5dea460c2b94d959d6f667de1a48773",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/694 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "087d7bcba0ba4f2f8034087a29cd8242",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "1_Pooling/config.json:   0%|          | 0.00/296 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name google-bert/bert-base-uncased. Creating a new one with mean pooling.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ec58c1fdcf04fd48d6cb22948f7048a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/570 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a14b1821949d45da90d4f5955e4c7ebf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/440M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77318d9f00e443e18708ce9a1457f872",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbad62e82ccf4a46aa7d7b717a04f3c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afefadd644ff47ccad645acdae3cd42b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name deepset/roberta-base-squad2. Creating a new one with mean pooling.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bcfc0cdb7c994b2aa639a1170e37dec9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/571 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e20b0f46922f42c2b76359cf39380595",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/496M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of RobertaModel were not initialized from the model checkpoint at deepset/roberta-base-squad2 and are newly initialized: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d86ded4a1b747f1a7609606d9368e4f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/79.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "847de608f72c42fab3c954b0aaa79e33",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0a4920ea1c1f4b86953467bfcdd95b82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d550bebe5a84b12853b6ea916da1db3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/772 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name albert/albert-base-v2. Creating a new one with mean pooling.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "994e1382f9744aa19b1d60c50497bdf0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/684 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47a0b16fee4a4eb3a187250a875d0f7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/47.4M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43952711f55743c392025a14b01f9918",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/25.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a221d52896234533a6e43edae2efdb70",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "spiece.model:   0%|          | 0.00/760k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12ababec540b41b39be575d66cdb7ea4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/1.31M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name distilbert/distilbert-base-uncased. Creating a new one with mean pooling.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5af7ba52959a49ec9d6ce318ddcc7481",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/483 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df7c9f909d16407393282a17af0764a1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9376a37a628f49aaaad8da3e43eb80f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13ea642d38994d3880b49551f1d19a2d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3051bd88387d469fb6cafec7f65b5913",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name Intel/dynamic_tinybert. Creating a new one with mean pooling.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b62c95a898c74effb14cdc4603d4b9ef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/743 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "025fde342e3044789e622405b22978de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of BertModel were not initialized from the model checkpoint at Intel/dynamic_tinybert and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Invalid model-index. Not loading eval results into CardData.\n",
      "WARNING:huggingface_hub.repocard_data:Invalid model-index. Not loading eval results into CardData.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39d98c0c95e34508a65cf475ef86a9db",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f8f8cee3d4be4c1b9d3b0218f2fb1f93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/351 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba54efe6c20b44bb9b7e68af4f445e39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3bbe9bb826dc42cb895e1eff21540f66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bca72e65e3340369e3e9f61a7781f4e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid model-index. Not loading eval results into CardData.\n",
      "WARNING:huggingface_hub.repocard_data:Invalid model-index. Not loading eval results into CardData.\n"
     ]
    }
   ],
   "source": [
    "# loading dataset and question answers\n",
    "rag_dataset = LabelledRagDataset.from_json(\"./data/rag_dataset.json\")\n",
    "documents = SimpleDirectoryReader(input_dir=\"./data/source_files\").load_data()\n",
    "\n",
    "\n",
    "# listed model to perform comparitive analysis for retrieval\n",
    "embed_models = {\n",
    "    \"modern_bert\": HuggingFaceEmbedding(model_name=\"nomic-ai/modernbert-embed-base\"),\n",
    "    \"google_bert\": HuggingFaceEmbedding(model_name=\"google-bert/bert-base-uncased\"),\n",
    "    \"roberta\": HuggingFaceEmbedding(model_name=\"deepset/roberta-base-squad2\"),\n",
    "    \"albert\": HuggingFaceEmbedding(model_name=\"albert/albert-base-v2\"),\n",
    "    \"distilbert\": HuggingFaceEmbedding(model_name=\"distilbert/distilbert-base-uncased\"),\n",
    "    \"tinybert\": HuggingFaceEmbedding(model_name=\"Intel/dynamic_tinybert\"),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 620,
     "referenced_widgets": [
      "07dfe246875148be84299bc2bff6e519",
      "9267e5cc24ef448d887be27135851439",
      "905ee3ddf99e4c7aa8914e24f7f84c2e",
      "d26605f8b2294c85ba7cd5c7b97315cf",
      "825808f829084682b310fcefa0b1c17a",
      "d81ffe0d84d440d8a371588420b76db8",
      "834e04552ced48fca299b3728ca01f0f",
      "8d286ba4105c41979b97504fdf67537a",
      "b713afa981ca4327a5f71bcccfddc929",
      "e2f6daa720f44580a770bbddb5f7dc47",
      "3d58a3a95a684f05a57b3fe2f9b5e3ae",
      "f657235aa4534d5ba7724d63e13f43d7",
      "eb95e82b17eb4cf9bf12a3db66e56a39",
      "8c720cf362dd4cdd859bf66b0f149196",
      "5ef83556459c46f78099de10348b7603",
      "dd75e4970e2c4789b4afd042e565da71",
      "72969cf7b7644fa8ab7a91e87360d49a",
      "8be391cb2e74406a8d9582e6e939d746",
      "b9835eb04f1745fd84c3fe44ee4fb6ef",
      "ec42a3f777114d0baad4ff01110bc95e",
      "853db254cfe048eeb3b8afc23fe23d47",
      "85332ed7a672411cb59d071fc4d05ab3",
      "d75a4ffdd9d249cba13116e19f6da820",
      "38892c5a05084d6191107cd55eb20bc9",
      "06bde860d94f4c7e9eee34a6b10a5ff5",
      "fd4324d9a83b401f9a4b38ba95341259",
      "242325f20a5246b29ce42f3fa63459af",
      "347a6cbe315b40f7b9f3e5a295c5633b",
      "20cddbca65bf450793411b4a4adc4719",
      "c8bfe5d77f2244699de38689cd550b9b",
      "1bdd894e4e0a4482a192314b47f689ce",
      "d0a71446aa36492bae51c0320dbf923d",
      "984fb07e042443258a199ca2972533b6",
      "0d500f7bb5124e11ad6c4f9f2ce79361",
      "e1991ebafafe492c8a91c7564513d82d",
      "570d97b29553472589d285c4e8392699",
      "f3f4df8586c44934bdc5cea1aef28ced",
      "3f077fa1cf1444c986a38306a87de7bd",
      "603ac53ccaab4321959e9e9ecacac945",
      "40c9d9576e564dd09718b53f0bb7ed59",
      "e75239c9c76b4b5eb446fc176fb5bd26",
      "32634e3a31a84122bb1ff3dcda0d52da",
      "1cf10e0e94b844988a7f6d615997a47b",
      "8f92b55405dd4e7a890b3db65550b645",
      "07349ba7050d4a939159dace7c809ee6",
      "0876eba9f8fb4e239a874afc49b50816",
      "f3636dcb8b424e9a98a24f65d0d4ff09",
      "f6c2b4582a764c51bf05eb44954839b6",
      "555b739d89f641a38a577f1c6dbcd610",
      "422cd9a367f44b64897d97e6e1bc3e72",
      "773b71e5db6b4b1e9bcc5ccdcd4f81b2",
      "536a3a07617d45b188fe5159935b6e2f",
      "ae4c9ce751b040bab8f60ef35f121c9d",
      "a6c8e1b5d48f4700aa7bc466c7e2a433",
      "3dcc7551b1664373a49487dfac489ab5",
      "b0e650f20d8749a2877c24b719131872",
      "38ec22fa355d46f7b3c34f23e7a274c8",
      "ee6c6984ec8841a7ac35bd0865d4dfac",
      "fd403cdfec3748eab8fd0cffe12c2fc4",
      "7ca67f03fc8146afbb559cf35815aa68",
      "4dbcbe71e0cf425f91f7f228931a5cc1",
      "fd6dfa1ef51d41299e916dfcc50064bc",
      "d0811a694f574b6ab7891ef29ccf6d46",
      "441d6c688b354f96828291cf0862bfd9",
      "f686b11916fd4f588acec823ce8da012",
      "a437a0d4414746709599167233fd016e",
      "d40b5d2cd6914fefb3d1709dd178afeb",
      "bcd51a4ff9c4454cbc1bcfdc9691cadb",
      "b784e0d098ad4acbbf19a3c1f08e6979",
      "ce8af2cb4b4e4e43942be54270186a2c",
      "f704098c02c94fc99f3a5087d1922d52",
      "c573329cd17f4adba418db365f07634c",
      "22b7ff3e666545efa51e8ce23586b550",
      "74199a635faf4d3fa667d6291d1cbe26",
      "34e9128f10d84385b8280f17a755d03f",
      "4ec61521bcf74ce6893eae1bb3b688a9",
      "31c81e0b6f2c454fa7eaf73fad09ade5",
      "9545ecf47e434b75868758850fb66be1",
      "28370796cd944b67aba28b376cdd9982",
      "8e11f42b41504ed5b637bcb001bed531",
      "74826821484d4a32a789e7c45d21e316",
      "575ea17c33154888a5f431c8bad1f472",
      "a074f146e61847eaaf559ae33a9f8349",
      "9950b82e7f564b929518bc165e1f2a3c",
      "92c82be8d7da4da484d6e11be1d7944e",
      "69df27ec29f847a9b05798aec61c6683",
      "acc188de937648a4a272530ac79459c0",
      "70aa986f261f4ecb9e9535c277c5ddc1",
      "16cd95b826584e2e95025f6c25bdc9fa",
      "dc0051c7282c44a099495baba0fd834f",
      "8516d4c33a404c3a91988f39a649968c",
      "1f15c2d60747455c9041e55e68ebde81",
      "deb1136eac764d2291856f3e2522da77",
      "ddaa99f0704c4f9b9b01451f8730abf6",
      "e263dd92c1004c8da1657870098c3d21",
      "72462794dbfc4affae151288866e3fae",
      "a0cab3d1024e44dea2ed2b1322c89a28",
      "5eadd3d2cfdc4cad9a7a23312f2d7cc1",
      "24662d3e14394e18967f7f914d81b2e0",
      "abb4915c1fe44a0a96168a1b70b1beca",
      "0de59830f25240949bd9a2b9636a43b2",
      "e8ae2e1f357d44fe890c533a415a0269",
      "b651461e5b7e46c7b0f69bacce0303fe",
      "a29ddbe20de64797b84359780e28a5ff",
      "e4035462890541479e040b1296ca454f",
      "e8acb522b70d4a689164762ccda8d12d",
      "cace2bb2cf394a3d8643e9642b70fdf3",
      "5d3adc71172f40a7bba9c89bc83e2c04",
      "6e3ebfb6f01947aba70fe342aadb20b9",
      "9f2b8a8a0c7f429f9297f37e189c1c77",
      "a4091fbad88344afa37c2b88ca9ba662",
      "62367e435f2f4cd4906621c4522eec82",
      "4b5535ef59b0465fae7e77a67c2d0653",
      "ca230ae9e27840b586c9b1f97eeab2d6",
      "614045388cf84960b785a5bc4f2530dd",
      "3ca045dc6e2c4e69b094798e4fc3a879",
      "20a2360723fb43db94c50aad159d5769",
      "7726ed65ceb4408993bf65b0620f8ec4",
      "5f66a6a5670f402fb6c0e9192ddfa1b7",
      "8518df0b79644777910784145a2b71b6",
      "bc450bba0f3a4aabb97f43c2d8aa007f",
      "e397b5846ff04b378570d0d84acd032c",
      "7098db99fc054e5c80eed0e5930b8f14",
      "7db6b6f8a88b4aecacba809d7f660664",
      "531e93414b1b43a9aad2cec81f0cc31a",
      "103542e590794a6b82577f237526fa24",
      "6cf0c8b974a644e7b8c3f6722c4acc29",
      "14e6529965ca4eacb46360d83634f713",
      "0a19d97397af40db8cbfe881d731b8d1",
      "d8c8b96aa6154f22bd74a671c23569ff",
      "546d8857298a4be6a8bb1e39d8363e2c",
      "ccc76bab8c384a97bcec17b93802cdee"
     ]
    },
    "id": "bi8nLAsHR7tG",
    "outputId": "9444ab36-8f21-49c3-ac2a-4ee3d599b20c"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07dfe246875148be84299bc2bff6e519",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/307 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f657235aa4534d5ba7724d63e13f43d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedder modern_bert: \n",
      "0.2749391727493917\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d75a4ffdd9d249cba13116e19f6da820",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/307 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d500f7bb5124e11ad6c4f9f2ce79361",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedder google_bert: \n",
      "0.20924574209245742\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07349ba7050d4a939159dace7c809ee6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/307 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0e650f20d8749a2877c24b719131872",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedder roberta: \n",
      "0.014598540145985401\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d40b5d2cd6914fefb3d1709dd178afeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/307 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9545ecf47e434b75868758850fb66be1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedder albert: \n",
      "0.16423357664233576\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16cd95b826584e2e95025f6c25bdc9fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/307 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "abb4915c1fe44a0a96168a1b70b1beca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedder distilbert: \n",
      "0.20802919708029197\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4091fbad88344afa37c2b88ca9ba662",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/307 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e397b5846ff04b378570d0d84acd032c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedder tinybert: \n",
      "0.012165450121654502\n"
     ]
    }
   ],
   "source": [
    "scores = {}\n",
    "\n",
    "# storing scores for comparison\n",
    "for embed_name, embed_model in embed_models.items():\n",
    "    eval_results = evaluate(\n",
    "        docs=documents,\n",
    "        dataset=rag_dataset,\n",
    "        embed_model=embed_model,\n",
    "        top_k=20,\n",
    "        verbose=False,\n",
    "    )\n",
    "    print(f\"Embedder {embed_name}: \")\n",
    "    score = pd.DataFrame(eval_results)[\"is_hit\"].mean()\n",
    "    print(score)\n",
    "    scores[embed_name] = score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "INH8wtJ__CGd",
    "outputId": "e9f844b8-61ff-4f49-9933-478246cc237a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'modern_bert': 0.2749391727493917,\n",
       " 'google_bert': 0.20924574209245742,\n",
       " 'roberta': 0.014598540145985401,\n",
       " 'albert': 0.16423357664233576,\n",
       " 'distilbert': 0.20802919708029197,\n",
       " 'tinybert': 0.012165450121654502}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1sX-cYSpJXkC"
   },
   "source": [
    "## Plotting Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ByyS0fTdJ4RX"
   },
   "source": [
    "Plotting evaluation scores to compare Modernbert with other models in bert series (Google's bert, Roberta, Albert, DistilBert, TinyBert) using QA dataset which is prepared using OpenAI embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 642
    },
    "id": "fUd_FjZG52el",
    "outputId": "c89da6f0-44a4-4e5e-d98a-419c854d5dd8"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-13-555835f472e3>:14: FutureWarning: \n",
      "\n",
      "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
      "\n",
      "  sns.barplot(x=\"Model\", y=\"Score\", data=df, palette=palette)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAICCAYAAAAzn+ywAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAYDxJREFUeJzt3XdYFFfbBvB7QWkiRUEQRREbalQEFVFjC4pg7L2Lij0Wosby2hIN9th7i0YUo4kae4nlU+zYYm9YomJBQUAB4fn+IDthBY0FWRju33Vx6Z6dWZ4ddnfuPXPmjEZEBEREREQqYaDvAoiIiIjSE8MNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww2Rnmk0GowdO/aD1wsLC4NGo8GKFSvSvSY1cnJyQpcuXfRdRppq1aqFL774Qt9lEKkGww0RgBUrVkCj0UCj0eDQoUOp7hcRODo6QqPR4Ouvv9ZDhZ8uPDwcgwcPhouLC8zMzJArVy64u7tj/PjxeP78ub7LowwQHR2NMWPG4IsvvkCuXLmQN29euLq6YsCAAbh//76+yyNKNzn0XQBRZmJiYoKgoCBUr15dp/3AgQO4d+8ejI2N9VTZpzlx4gR8fX0RHR2NDh06wN3dHQBw8uRJTJw4EQcPHsSuXbv0XOXndeXKFRgYZN/vcwkJCahRowYuX76Mzp0745tvvkF0dDQuXLiAoKAgNG3aFA4ODvoukyhdMNwQpeDr64tff/0Vs2bNQo4c/749goKC4O7ujidPnuixuo/z/PlzNG3aFIaGhjh9+jRcXFx07p8wYQIWL16sp+o+LxHBq1evYGpqmmWDaXrZuHEjTp8+jdWrV6Ndu3Y697169Qrx8fEZVktMTAxy5cqVYb+Psp/s+zWGKA1t27bF06dPsXv3bqUtPj4e69evT7VD0IqJicG3334LR0dHGBsbo2TJkpg6dSpERGe5uLg4DBo0CLa2tsidOzcaNWqEe/fupfmYf//9N7p27Qo7OzsYGxujTJkyWLZs2Uc9p4ULF+Lvv//G9OnTUwUbALCzs8P//vc/nbZ58+ahTJkyMDY2hoODA/r27Zvq0JV2nMi5c+dQs2ZNmJmZoVixYli/fj2A5N4uDw8PmJqaomTJktizZ4/O+mPHjoVGo8Hly5fRqlUrWFhYIG/evBgwYABevXqls+zy5ctRp04d5MuXD8bGxihdujTmz5+f6rk4OTnh66+/xs6dO1GxYkWYmppi4cKFyn0px9wkJCRg3LhxKF68OExMTJA3b15Ur15d528PAH/++Se+/PJL5MqVC1ZWVmjcuDEuXbqU5nO5fv06unTpAisrK1haWsLPzw+xsbFp/FXSdurUKVStWhWmpqYoUqQIFixYoNwXHR2NXLlyYcCAAanWu3fvHgwNDREYGPjWx75x4wYAoFq1aqnuMzExgYWFhU6b9u9ia2ur/A1Hjhyps8zp06fh4+MDCwsLmJub46uvvsLRo0d1ltEe8j1w4AD69OmDfPnyoWDBgsr927dvV7Zv7ty50aBBA1y4cOEdW4novzHcEKXg5OQET09PrFmzRmnbvn07IiMj0aZNm1TLiwgaNWqEn376CfXr18f06dNRsmRJDBkyBAEBATrLdu/eHTNmzEC9evUwceJE5MyZEw0aNEj1mOHh4ahSpQr27NmDfv36YebMmShWrBi6deuGGTNmfPBz2rx5M0xNTdGiRYv3Wn7s2LHo27cvHBwcMG3aNDRv3hwLFy5EvXr1kJCQoLPss2fP8PXXX8PDwwOTJ0+GsbEx2rRpg+DgYLRp0wa+vr6YOHEiYmJi0KJFC7x48SLV72vVqhVevXqFwMBA+Pr6YtasWejRo4fOMvPnz0fhwoUxYsQITJs2DY6OjujTpw/mzp2b6vGuXLmCtm3bom7dupg5cyZcXV3f+jzHjRuH2rVrY86cORg5ciQKFSqE0NBQZZk9e/bA29sbjx49wtixYxEQEICQkBBUq1YNYWFhaT6XFy9eIDAwEK1atcKKFSswbty499jqydvS19cX7u7umDx5MgoWLIjevXsrodbc3BxNmzZFcHAwEhMTddZds2YNRATt27d/6+MXLlwYALBy5cpUwftN586dg4eHB/7880/4+/tj5syZaNKkCf744w9lmQsXLuDLL7/E2bNnMXToUIwaNQq3bt1CrVq1cOzYsVSP2adPH1y8eBGjR4/GsGHDAACrVq1CgwYNYG5ujkmTJmHUqFG4ePEiqlevnub2JXpvQkSyfPlyASAnTpyQOXPmSO7cuSU2NlZERFq2bCm1a9cWEZHChQtLgwYNlPU2btwoAGT8+PE6j9eiRQvRaDRy/fp1ERE5c+aMAJA+ffroLNeuXTsBIGPGjFHaunXrJvnz55cnT57oLNumTRuxtLRU6rp165YAkOXLl7/zuVlbW0v58uXfazs8evRIjIyMpF69epKYmKi0z5kzRwDIsmXLlLaaNWsKAAkKClLaLl++LADEwMBAjh49qrTv3LkzVa1jxowRANKoUSOdGvr06SMA5OzZs0qb9jmn5O3tLc7OzjpthQsXFgCyY8eOVMsXLlxYOnfurNwuX768zt8yLa6urpIvXz55+vSp0nb27FkxMDCQTp06pXouXbt21Vm/adOmkjdv3nf+DpF/t+W0adOUtri4OOX3x8fHi8i/23H79u0665crV05q1qz5zt8RGxsrJUuWFABSuHBh6dKliyxdulTCw8NTLVujRg3JnTu33L59W6c9KSlJ+X+TJk3EyMhIbty4obTdv39fcufOLTVq1FDatO+t6tWry+vXr5X2Fy9eiJWVlfj7++v8jocPH4qlpWWqdqIPwZ4boje0atUKL1++xJYtW/DixQts2bLlrYektm3bBkNDQ/Tv31+n/dtvv4WIYPv27cpyAFItN3DgQJ3bIoINGzagYcOGEBE8efJE+fH29kZkZKROz8L7iIqKQu7cud9r2T179iA+Ph4DBw7UGXzr7+8PCwsLbN26VWd5c3NznR6tkiVLwsrKCqVKlYKHh4fSrv3/zZs3U/3Ovn376tz+5ptvAPy7zQDA1NRU+X9kZCSePHmCmjVr4ubNm4iMjNRZv0iRIvD29v7P52plZYULFy7g2rVrad7/4MEDnDlzBl26dEGePHmU9nLlyqFu3bo69Wn16tVL5/aXX36Jp0+fIioq6j/ryZEjB3r27KncNjIyQs+ePfHo0SOcOnUKAODl5QUHBwesXr1aWe6vv/7CuXPn0KFDh3c+vqmpKY4dO4YhQ4YASD5c1K1bN+TPnx/ffPMN4uLiAACPHz/GwYMH0bVrVxQqVEjnMTQaDQAgMTERu3btQpMmTeDs7Kzcnz9/frRr1w6HDh1K9Zz9/f1haGio3N69ezeeP3+Otm3b6rzODQ0N4eHhgX379v3nNiN6G4YbojfY2trCy8sLQUFB+O2335CYmPjWQzq3b9+Gg4NDqvBQqlQp5X7tvwYGBihatKjOciVLltS5/fjxYzx//hyLFi2Cra2tzo+fnx8A4NGjRx/0fCwsLNI8HPS255NWXUZGRnB2dlbu1ypYsKCyw9OytLSEo6NjqjYg+dDLm4oXL65zu2jRojAwMNA5LHH48GF4eXkp415sbW0xYsQIAEgz3LyP77//Hs+fP0eJEiVQtmxZDBkyBOfOnVPuf9u2AJL/vk+ePEFMTIxO+5thwNraGkDaz/tNDg4OqQbZlihRAgCUbWFgYID27dtj48aNylie1atXw8TEBC1btvzP32FpaYnJkycjLCwMYWFhWLp0KUqWLIk5c+bghx9+APBvAH3XvDuPHz9GbGzsW7dNUlIS7t69q9P+5t9FGyrr1KmT6rW+a9euD36dE6XEs6WI0tCuXTv4+/vj4cOH8PHxgZWVVYb83qSkJABAhw4d0Llz5zSXKVeu3Ac9pouLC86cOYP4+HgYGRl9co0ppfwm/j7t8h9jPQCkCks3btzAV199BRcXF0yfPh2Ojo4wMjLCtm3b8NNPPynbTCtlL8+71KhRAzdu3MCmTZuwa9cuLFmyBD/99BMWLFiA7t27v9djvOlTnvf76tSpE6ZMmYKNGzeibdu2CAoKwtdff60EyPdVuHBhdO3aFU2bNoWzszNWr16N8ePHp1udb3rz76L9u61atQr29vaplk95tiLRh+KrhygNTZs2Rc+ePXH06FEEBwe/dbnChQtjz549ePHihU7vzeXLl5X7tf8mJSXhxo0bOt92r1y5ovN42jOpEhMT4eXllS7PpWHDhjhy5Ag2bNiAtm3bvnNZbb1XrlzROdwQHx+PW7dupVtNKV27dk3nW/3169eRlJQEJycnAMAff/yBuLg4bN68WadnJD0OW+TJkwd+fn7w8/NDdHQ0atSogbFjx6J79+462+JNly9fho2NTbqeznz//v1Up0hfvXoVAJRtAST3qFSoUAGrV69GwYIFcefOHcyePfujf6+1tTWKFi2Kv/76CwCUv7v2dlpsbW1hZmb21m1jYGCQqvfuTdpezHz58n2W1xVlbzwsRZQGc3NzzJ8/H2PHjkXDhg3fupyvry8SExMxZ84cnfaffvoJGo0GPj4+AKD8O2vWLJ3l3jz7ydDQEM2bN8eGDRvS3Lk8fvz4g59Lr169kD9/fnz77bfKzjKlR48eKd/Yvby8YGRkhFmzZun0NixduhSRkZFpnt31qd4840m7o9ZuM21vSMp6IiMjsXz58k/6vU+fPtW5bW5ujmLFiiljT/Lnzw9XV1f8/PPPOqfB//XXX9i1axd8fX0/6fe/6fXr18pp60ByoFy4cCFsbW2VSRe1OnbsiF27dmHGjBnImzevsq3e5ezZs2nO03T79m1cvHhRCd22traoUaMGli1bhjt37ugsq/0bGBoaol69eti0aZPO4cPw8HBlEsw3Ty1/k7e3NywsLPDjjz+mOgsP+LjXOpEWe26I3uJth4VSatiwIWrXro2RI0ciLCwM5cuXx65du7Bp0yYMHDhQ+Xbq6uqKtm3bYt68eYiMjETVqlWxd+9eXL9+PdVjTpw4Efv27YOHhwf8/f1RunRpREREIDQ0FHv27EFERMQHPQ9ra2v8/vvv8PX1haurq84MxaGhoVizZg08PT0BJO/Yhg8fjnHjxqF+/fpo1KgRrly5gnnz5qFSpUr/OWj1Y9y6dQuNGjVC/fr1ceTIEfzyyy9o164dypcvDwCoV68ejIyM0LBhQ/Ts2RPR0dFYvHgx8uXLhwcPHnz07y1dujRq1aoFd3d35MmTBydPnsT69evRr18/ZZkpU6bAx8cHnp6e6NatG16+fInZs2fD0tLyo64H9i4ODg6YNGkSwsLCUKJECQQHB+PMmTNYtGgRcubMqbNsu3btMHToUPz+++/o3bt3qvvTsnv3bowZMwaNGjVClSpVYG5ujps3b2LZsmWIi4vTeT6zZs1C9erV4ebmhh49eqBIkSIICwvD1q1bcebMGQDA+PHjsXv3blSvXh19+vRBjhw5sHDhQsTFxWHy5Mn/WY+FhQXmz5+Pjh07ws3NDW3atIGtrS3u3LmDrVu3olq1aqm+NBC9N/2dqEWUeaQ8Ffxd3jwVXCT5lNZBgwaJg4OD5MyZU4oXLy5TpkzROW1WROTly5fSv39/yZs3r+TKlUsaNmwod+/eTXUquIhIeHi49O3bVxwdHSVnzpxib28vX331lSxatEhZ5n1PBde6f/++DBo0SEqUKCEmJiZiZmYm7u7uMmHCBImMjNRZds6cOeLi4iI5c+YUOzs76d27tzx79kxnmZo1a0qZMmXeaxuJiACQvn37Kre1p09fvHhRWrRoIblz5xZra2vp16+fvHz5UmfdzZs3S7ly5cTExEScnJxk0qRJsmzZMgEgt27d+s/frb0v5ang48ePl8qVK4uVlZWYmpqKi4uLTJgwQTntWmvPnj1SrVo1MTU1FQsLC2nYsKFcvHhRZxntc3n8+LFOu/Z1lbLGtGi35cmTJ8XT01NMTEykcOHCMmfOnLeu4+vrKwAkJCTknY+tdfPmTRk9erRUqVJF8uXLJzly5BBbW1tp0KCB/Pnnn6mW/+uvv6Rp06ZiZWUlJiYmUrJkSRk1apTOMqGhoeLt7S3m5uZiZmYmtWvXTlXPf7239u3bJ97e3mJpaSkmJiZStGhR6dKli5w8efK9nhdRWjQi6TjSjYjoPWkn0Xv8+DFsbGz0XU6W07RpU5w/fz7N3j+i7I5jboiIspgHDx5g69at6Nixo75LIcqUOOaGiCiLuHXrFg4fPowlS5YgZ86cOpP+EdG/2HNDRJRFHDhwAB07dsStW7fw888/pzk/DBEBmWLMzdy5czFlyhQ8fPgQ5cuXx+zZs1G5cuU0l12xYoUyU6uWsbFxqqsIExERUfak956b4OBgBAQEYMyYMQgNDUX58uWVq/C+jYWFBR48eKD8vDklPBEREWVfeg8306dPh7+/P/z8/FC6dGksWLAAZmZmWLZs2VvX0Wg0sLe3V37s7OwysGIiIiLKzPQ6oDg+Ph6nTp3C8OHDlTYDAwN4eXnhyJEjb10vOjpamc7ezc0NP/74I8qUKZPmsnFxccqMo0Dy9UwiIiKQN2/eVNewISIiosxJRPDixQs4ODjAwODdfTN6DTdPnjxBYmJiqp4XOzs75do8bypZsiSWLVuGcuXKITIyElOnTkXVqlVx4cIFFCxYMNXygYGBGDdu3Gepn4iIiDLW3bt309zfp5TlTgX39PRUpooHgKpVq6JUqVJYuHAhfvjhh1TLDx8+HAEBAcrtyMhIFCpUCHfv3v3Pa58QERFR5hAVFQVHR0edixS/jV7DjY2NDQwNDREeHq7THh4e/t6nOObMmRMVKlR46yydxsbGMDY2TtVuYWHBcENERJTFvM+QEr0OKDYyMoK7uzv27t2rtCUlJWHv3r06vTPvkpiYiPPnzyN//vyfq0wiIiLKQvR+WCogIACdO3dGxYoVUblyZcyYMQMxMTHKXDadOnVCgQIFEBgYCAD4/vvvUaVKFRQrVgzPnz/HlClTcPv2bXTv3l2fT4OIiIgyCb2Hm9atW+Px48cYPXo0Hj58CFdXV+zYsUMZZHznzh2dUdHPnj2Dv78/Hj58CGtra7i7uyMkJASlS5fW11MgIiKiTCRTzFCckaKiomBpaYnIyEiOuSEiIsoiPmT/rfdJ/IiIiIjSE8MNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREakKww0RERGpCsMNERERqQrDDREREalKpgg3c+fOhZOTE0xMTODh4YHjx4+/13pr166FRqNBkyZNPm+BRERElGXoPdwEBwcjICAAY8aMQWhoKMqXLw9vb288evToneuFhYVh8ODB+PLLLzOoUiIiIsoK9B5upk+fDn9/f/j5+aF06dJYsGABzMzMsGzZsreuk5iYiPbt22PcuHFwdnbOwGqJiIgos9NruImPj8epU6fg5eWltBkYGMDLywtHjhx563rff/898uXLh27duv3n74iLi0NUVJTODxEREamXXsPNkydPkJiYCDs7O512Ozs7PHz4MM11Dh06hKVLl2Lx4sXv9TsCAwNhaWmp/Dg6On5y3URERJR56f2w1Id48eIFOnbsiMWLF8PGxua91hk+fDgiIyOVn7t3737mKomIiEifcujzl9vY2MDQ0BDh4eE67eHh4bC3t0+1/I0bNxAWFoaGDRsqbUlJSQCAHDly4MqVKyhatKjOOsbGxjA2Nv4M1RMREVFmpNeeGyMjI7i7u2Pv3r1KW1JSEvbu3QtPT89Uy7u4uOD8+fM4c+aM8tOoUSPUrl0bZ86c4SEnIiIi0m/PDQAEBASgc+fOqFixIipXrowZM2YgJiYGfn5+AIBOnTqhQIECCAwMhImJCb744gud9a2srAAgVTsRERFlT3oPN61bt8bjx48xevRoPHz4EK6urtixY4cyyPjOnTswMMhSQ4OIiIhIjzQiIvouIiNFRUXB0tISkZGRsLCw0Hc5RERE9B4+ZP/NLhEiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlKVTBFu5s6dCycnJ5iYmMDDwwPHjx9/67K//fYbKlasCCsrK+TKlQuurq5YtWpVBlZLREREmZnew01wcDACAgIwZswYhIaGonz58vD29sajR4/SXD5PnjwYOXIkjhw5gnPnzsHPzw9+fn7YuXNnBldOREREmZFGRESfBXh4eKBSpUqYM2cOACApKQmOjo745ptvMGzYsPd6DDc3NzRo0AA//PDDfy4bFRUFS0tLREZGwsLC4pNqJyIioozxIftvvfbcxMfH49SpU/Dy8lLaDAwM4OXlhSNHjvzn+iKCvXv34sqVK6hRo0aay8TFxSEqKkrnh4iIiNRLr+HmyZMnSExMhJ2dnU67nZ0dHj58+Nb1IiMjYW5uDiMjIzRo0ACzZ89G3bp101w2MDAQlpaWyo+jo2O6PgciIiLKXPQ+5uZj5M6dG2fOnMGJEycwYcIEBAQEYP/+/WkuO3z4cERGRio/d+/ezdhiiYiIKEPl0Ocvt7GxgaGhIcLDw3Xaw8PDYW9v/9b1DAwMUKxYMQCAq6srLl26hMDAQNSqVSvVssbGxjA2Nk7XuomIiCjz0mvPjZGREdzd3bF3716lLSkpCXv37oWnp+d7P05SUhLi4uI+R4lERESUxei15wYAAgIC0LlzZ1SsWBGVK1fGjBkzEBMTAz8/PwBAp06dUKBAAQQGBgJIHkNTsWJFFC1aFHFxcdi2bRtWrVqF+fPn6/NpEBERUSah93DTunVrPH78GKNHj8bDhw/h6uqKHTt2KIOM79y5AwODfzuYYmJi0KdPH9y7dw+mpqZwcXHBL7/8gtatW+vrKRAREVEmovd5bjIa57khIiLKerLMPDdERERE6Y3hhoiIiFSF4YaIiIhUheGGiIiIVIXhhoiIiFSF4YaIiIhUheGGiIiIVIXhhoiIiFSF4YaIiIhUheGGiIiIVIXhhoiIiFRF7xfOzKzajd6v7xIyhaDva+m7BCIiog/CnhsiIiJSlU8KN/Hx8bhy5Qpev36dXvUQERERfZKPCjexsbHo1q0bzMzMUKZMGdy5cwcA8M0332DixInpWiARERHRh/iocDN8+HCcPXsW+/fvh4mJidLu5eWF4ODgdCuOiIiI6EN91IDijRs3Ijg4GFWqVIFGo1Hay5Qpgxs3bqRbcUREREQf6qN6bh4/fox8+fKlao+JidEJO0REREQZ7aPCTcWKFbF161bltjbQLFmyBJ6enulTGREREdFH+KjDUj/++CN8fHxw8eJFvH79GjNnzsTFixcREhKCAwcOpHeNRERERO/to3puqlevjrNnz+L169coW7Ysdu3ahXz58uHIkSNwd3dP7xqJiIiI3tsH99wkJCSgZ8+eGDVqFBYvXvw5aiIiIiL6aB/cc5MzZ05s2LDhc9RCRERE9Mk+6rBUkyZNsHHjxnQuhYiIiOjTfdSA4uLFi+P777/H4cOH4e7ujly5cunc379//3QpjoiIiOhDfVS4Wbp0KaysrHDq1CmcOnVK5z6NRsNwQ0RERHrzUeHm1q1b6V0HERERUbr4pKuCA4CIQETSoxYiIiKiT/bR4WblypUoW7YsTE1NYWpqinLlymHVqlXpWRsRERHRB/uow1LTp0/HqFGj0K9fP1SrVg0AcOjQIfTq1QtPnjzBoEGD0rVIIiIiovf1UeFm9uzZmD9/Pjp16qS0NWrUCGXKlMHYsWMZboiIiEhvPuqw1IMHD1C1atVU7VWrVsWDBw8+uSgiIiKij/VR4aZYsWJYt25dqvbg4GAUL178k4siIiIi+lgfdVhq3LhxaN26NQ4ePKiMuTl8+DD27t2bZughIiIiyigf1XPTvHlzHDt2DDY2Nti4cSM2btwIGxsbHD9+HE2bNk3vGomIiIje20f13ACAu7s7fvnll/SshYiIiOiTfVTPzbZt27Bz585U7Tt37sT27ds/uSgiIiKij/VR4WbYsGFITExM1S4iGDZs2CcXRURERPSxPircXLt2DaVLl07V7uLiguvXr39yUUREREQf66PCjaWlJW7evJmq/fr168iVK9cnF0VERET0sT4q3DRu3BgDBw7EjRs3lLbr16/j22+/RaNGjdKtOCIiIqIP9VHhZvLkyciVKxdcXFxQpEgRFClSBC4uLsibNy+mTp2a3jUSERERvbePOhXc0tISISEh2L17N86ePQtTU1OUL18eX375ZXrXR0RERPRBPqjn5siRI9iyZQsAQKPRoF69esiXLx+mTp2K5s2bo0ePHoiLi/sshRIRERG9jw8KN99//z0uXLig3D5//jz8/f1Rt25dDBs2DH/88QcCAwPTvUgiIiKi9/VB4ebMmTP46quvlNtr165F5cqVsXjxYgQEBGDWrFm8thQRERHp1QeFm2fPnsHOzk65feDAAfj4+Ci3K1WqhLt376ZfdUREREQf6IPCjZ2dHW7dugUAiI+PR2hoKKpUqaLc/+LFC+TMmTN9KyQiIiL6AB8Ubnx9fTFs2DD83//9H4YPHw4zMzOdM6TOnTuHokWLpnuRRERERO/rg04F/+GHH9CsWTPUrFkT5ubm+Pnnn2FkZKTcv2zZMtSrVy/diyQiIiJ6Xx8UbmxsbHDw4EFERkbC3NwchoaGOvf/+uuvMDc3T9cCiYiIiD7ER0/il5Y8efJ8UjFEREREn+qjLr9ARERElFkx3BAREZGqMNwQERGRqjDcEBERkaow3BAREZGqMNwQERGRqjDcEBERkaow3BAREZGqfNQkfkTv6+rULvouIVMoMXiFvksgIso2GG6Isoih+wP0XUKmMLnWdH2XQESZHA9LERERkapkinAzd+5cODk5wcTEBB4eHjh+/Phbl128eDG+/PJLWFtbw9raGl5eXu9cnoiIiLIXvYeb4OBgBAQEYMyYMQgNDUX58uXh7e2NR48epbn8/v370bZtW+zbtw9HjhyBo6Mj6tWrh7///juDKyciIqLMSO/hZvr06fD394efnx9Kly6NBQsWwMzMDMuWLUtz+dWrV6NPnz5wdXWFi4sLlixZgqSkJOzduzeDKyciIqLMSK/hJj4+HqdOnYKXl5fSZmBgAC8vLxw5cuS9HiM2NhYJCQnIkydPmvfHxcUhKipK54eIiIjUS6/h5smTJ0hMTISdnZ1Ou52dHR4+fPhej/Hdd9/BwcFBJyClFBgYCEtLS+XH0dHxk+smIiKizEvvh6U+xcSJE7F27Vr8/vvvMDExSXOZ4cOHIzIyUvm5e/duBldJREREGUmv89zY2NjA0NAQ4eHhOu3h4eGwt7d/57pTp07FxIkTsWfPHpQrV+6tyxkbG8PY2Dhd6iUiIqLMT689N0ZGRnB3d9cZDKwdHOzp6fnW9SZPnowffvgBO3bsQMWKFTOiVCIiIsoi9D5DcUBAADp37oyKFSuicuXKmDFjBmJiYuDn5wcA6NSpEwoUKIDAwEAAwKRJkzB69GgEBQXByclJGZtjbm4Oc3NzvT0PIiIiyhz0Hm5at26Nx48fY/To0Xj48CFcXV2xY8cOZZDxnTt3YGDwbwfT/PnzER8fjxYtWug8zpgxYzB27NiMLJ2IiIgyIb2HGwDo168f+vXrl+Z9+/fv17kdFhb2+QsiIiKiLCtLny1FRERE9CaGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUhWGGyIiIlIVhhsiIiJSFYYbIiIiUpUc+i6AiCijHR0wQN8lZApVZs7UdwlEnwV7boiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhVGG6IiIhIVRhuiIiISFUYboiIiEhV9B5u5s6dCycnJ5iYmMDDwwPHjx9/67IXLlxA8+bN4eTkBI1GgxkzZmRcoURERJQl6DXcBAcHIyAgAGPGjEFoaCjKly8Pb29vPHr0KM3lY2Nj4ezsjIkTJ8Le3j6DqyUiIqKsQK/hZvr06fD394efnx9Kly6NBQsWwMzMDMuWLUtz+UqVKmHKlClo06YNjI2NM7haIiIiygr0Fm7i4+Nx6tQpeHl5/VuMgQG8vLxw5MiRdPs9cXFxiIqK0vkhIiIi9dJbuHny5AkSExNhZ2en025nZ4eHDx+m2+8JDAyEpaWl8uPo6Jhuj01ERESZj94HFH9uw4cPR2RkpPJz9+5dfZdEREREn1EOff1iGxsbGBoaIjw8XKc9PDw8XQcLGxsbc3wOERFRNqK3nhsjIyO4u7tj7969SltSUhL27t0LT09PfZVFREREWZzeem4AICAgAJ07d0bFihVRuXJlzJgxAzExMfDz8wMAdOrUCQUKFEBgYCCA5EHIFy9eVP7/999/48yZMzA3N0exYsX09jyIiIgo89BruGndujUeP36M0aNH4+HDh3B1dcWOHTuUQcZ37tyBgcG/nUv3799HhQoVlNtTp07F1KlTUbNmTezfvz+jyyciIqJMSK/hBgD69euHfv36pXnfm4HFyckJIpIBVREREVFWpfqzpYiIiCh7YbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVWG4ISIiIlVhuCEiIiJVYbghIiIiVcmh7wKIiChrmjt0vb5LyBT6Tm6h7xLoDey5ISIiIlVhzw0REZGe/dDNW98lZAqjlu5Ml8dhzw0RERGpCsMNERERqQrDDREREalKpgg3c+fOhZOTE0xMTODh4YHjx4+/c/lff/0VLi4uMDExQdmyZbFt27YMqpSIiIgyO72Hm+DgYAQEBGDMmDEIDQ1F+fLl4e3tjUePHqW5fEhICNq2bYtu3brh9OnTaNKkCZo0aYK//vorgysnIiKizEjv4Wb69Onw9/eHn58fSpcujQULFsDMzAzLli1Lc/mZM2eifv36GDJkCEqVKoUffvgBbm5umDNnTgZXTkRERJmRXsNNfHw8Tp06BS8vL6XNwMAAXl5eOHLkSJrrHDlyRGd5APD29n7r8kRERJS96HWemydPniAxMRF2dnY67XZ2drh8+XKa6zx8+DDN5R8+fJjm8nFxcYiLi1NuR0ZGAgCioqLeWVtCXMx/1p8d/Nd2+i/Rr+LTqZKs7VO3IwDExcT990LZQHpsy5g4bkvg07fly7jYdKoka0uP1+Sr+NfpUEnW965tqb1PRP7zcVQ/iV9gYCDGjRuXqt3R0VEP1WQ96yfruwKVGLVG3xWoxizM03cJ6rFwob4rUIUhs/RdgXr8+Ivlfy7z4sULWFq+ezm9hhsbGxsYGhoiPDxcpz08PBz29vZprmNvb/9Byw8fPhwBAQHK7aSkJERERCBv3rzQaDSf+Aw+n6ioKDg6OuLu3buwsLDQdzlZFrdj+uG2TD/clumD2zH9ZIVtKSJ48eIFHBwc/nNZvYYbIyMjuLu7Y+/evWjSpAmA5PCxd+9e9OvXL811PD09sXfvXgwcOFBp2717Nzw9PdNc3tjYGMbGxjptVlZW6VF+hrCwsMi0L7SshNsx/XBbph9uy/TB7Zh+Mvu2/K8eGy29H5YKCAhA586dUbFiRVSuXBkzZsxATEwM/Pz8AACdOnVCgQIFEBgYCAAYMGAAatasiWnTpqFBgwZYu3YtTp48iUWLFunzaRAREVEmofdw07p1azx+/BijR4/Gw4cP4erqih07diiDhu/cuQMDg39P6qpatSqCgoLwv//9DyNGjEDx4sWxceNGfPHFF/p6CkRERJSJ6D3cAEC/fv3eehhq//79qdpatmyJli1bfuaq9MvY2BhjxoxJdUiNPgy3Y/rhtkw/3Jbpg9sx/ahtW2rkfc6pIiIiIsoi9D5DMREREVF6YrghIiIiVWG4ISIiIlVhuCEiIiJVYbihLItj4YmI3i4xMVHndnb6zGS4yWDZ6cX1uWkvn6G9GCq9v6SkpFRtz58/z/hCVIbvb8oskpKSYGhoCADYt28fEhMTM/Ulh9Ibw00GSkpKylYvrs9l165dCAkJAQAMHToUCxcuxOvXvKLuhzAwMMCtW7ewYMECAMC6devQr18/RERE6LmyrEUbEl+9egUgOXCHhITg/v37+iyLsrnNmzfD19cXADBo0CB8++232e69nSkm8csOjh49ikKFCsHBwQGDBw9GkSJF0LdvX32XleU8fvwYc+fOxY0bN1C+fHmsX78eJ06cQI4cfCl/iISEBCxfvhzLli3DqVOnsHTpUixfvhx58uTRd2lZioGBAe7duwcfHx9s27YNR48eRefOnbF169b3urgfUXoTEVhZWeHYsWMoVaoU7t+/j6NHj8LW1lbfpWUoTuL3mSUlJeHp06ews7NDx44dYWxsjHXr1uHw4cMoU6aMvsvLUkQEGo0Gx48fR4sWLfDgwQMsXboUnTp1QmJiotIFS++m3Y73799H79698ccff6B9+/ZYtWoVgOTXbMpLntC7xcTEoHHjxrhw4QIePXqE5cuXo1OnTvouK0vSvjafPn2KHDlyICkpCdbW1vouK8tI+TnYqlUrrF+/HjVq1FBm+s9O7+3s8Sz1zNbWFhcuXMCvv/6KFStWYN26dQw2H0j7oadVpUoV1KhRA7NmzcKhQ4dgaGgIEeGYh/eg3UYxMTEoW7Ys6tWrh9OnT+Onn34CkNwb8eZAREpbUlIScuXKhW7duiE8PBxWVlaoUqWKvsvKkrTv8S1btqBly5aoXLkyunTpgjVr1ui7tCxDG2zmz5+PHDlyYMmSJbh27RoaNmwIIPm9/eYhfNV+Zgp9NklJScr/Fy9eLE5OTmJkZCR+fn5y+/btNJej1BITE5X/9+/fX0qVKiV37tyRQ4cOSbNmzcTV1VUOHTqks87jx48zuswsZePGjaLRaOTo0aNy7do1GTx4sJQsWVKmT5+us9z9+/f1VGHWsWPHDunQoYMsWbJEGjVqJAUKFJCTJ0/qu6wsafPmzWJmZiaTJ0+W3377TXr37i1mZmaybNkyfZeWqaX8jJw/f77Y2NjIX3/9JSIi+/btE3t7e/n666911tmwYYO8fv06Q+vMSAw3n0nKwDJy5EipXbu2PHjwQM6ePSvGxsbSoUMHuXPnjh4rzHoiIiKkQ4cO8ueffyptBw4ckObNm4u7u7scPHhQRESaN28us2fP1leZmd7z589lypQpMm3aNKXt1q1bMmTIEClVqpTSPnr0aOnUqZPExsbqq9RM78yZM5I3b14JDg4WEZGYmBipX7++FChQQE6fPq0sFxwczKD4H27evCmenp4yd+5cERF59OiRFCxYUFxdXSVXrlyyZMkSPVeY+R09elT+97//yYoVK3Ta9+/fL/nz55d69erJyZMnpV69evLVV1/phCK1Ybj5zI4dOyZfffWVHDlyRAk8ISEhYmxsLJ07d5abN2+KiEizZs347eQdtN9GqlSpomwzrYMHD0rr1q0ld+7cUrFiRXFycpL4+Hg9VZq5acN16dKl5bffftO57+bNmzJixAixtrYWNzc3yZ07txw/flxPlWZ+Fy5ckClTpsjAgQNFRCQhIUFEkgOOj4+PODg4SFBQkAwaNEisrKzkxo0b+iw300i5Q03Zc/DkyRMJCAiQBw8eyL1796RkyZLSs2dPCQsLE29vb8mZM6fMmzdPHyVneklJSXLu3DnRaDSi0WiUgJjS8ePHpWjRolKyZEnx9PRUPiPVeuSA4eYzWrlypTRp0kR8fX0lLi5OXr9+rXwAhoSEiLm5uVSvXl3Kly8vJUuW5A75HY4dOyYeHh6SK1cuuXTpkoiIzva6cuWKrFq1SsaPH69sY+2/9K8HDx5I586dRaPRKN+EU26nR48eye7duyUwMFCuXr2qrzIztaSkJImOjhYXFxfRaDTSuHFj5T7tjjshIUFatGghxYsXlzJlysipU6f0VG3mdO/ePTl79qyIiKxfv14WL14sIiKRkZEiIvLdd99J06ZN5fnz5yIiMnDgQClYsKAULlxYIiIiVLtD/hDabZByW/z666+i0WikVatW8vfff6da5+XLl3Lq1Cmd16laMdx8RuPGjZP8+fNL/vz5lW9tiYmJygvq9OnTMmLECBk1ahR3yCm8rav0zJkzUrJkSalYsaK8fPlSRN6+vdR8LPlDpLUTePLkibRr105y5colISEhIsLt9b5Sbs9Lly5JpUqVxNnZWbZv3660p9yW165dk4iIiAytMbOLioqSJk2aiI+Pj0ydOlU0Go2sWrVKuT8xMVHq1asnfn5+Sts333wj8+bNk2fPnumh4swn5WdkdHS0xMXFKZ+FK1euFI1GI9999508efJEWe7N97iaD0mJMNykm7e9UObPny9FihQRf39/5XBKUlKS8kJLuR6Dje722LJli8yePVtWrlwpZ86cEZHkwyrFixeXKlWqyKtXr0SEO+a30e6Ijx49KgsXLpQJEyYo45JiY2OldevWYm5uLkePHhUR9X/YfQrttnz+/LnExcUpgeXSpUvyxRdfiI+Pj7JtRYS9sP9h48aNUrZsWdFoNDJ+/HgRSX79abfzDz/8IIUKFZLAwEDp06eP5MuXj4f1/pHyfTp9+nRp1KiR1K5dWzp27ChPnz4VkX8DzrBhw5S27IbhJh2kfLHt3r1bNm/erPNNZObMmeLq6ioDBgyQW7duiYh6j3OmlyFDhkjBggWlfv368uWXX4qLi4sEBQWJiEhoaKiULFlSqlatqvTgUNrWr18vVlZW0rp1a6lSpYq4ublJnz59REQkPDxc2rZtK9bW1qnONqN/ad+rW7duFW9vb6lSpYpUqVJFdu3aJSLJAadMmTLi6+sr//d//6fPUjM97Re4U6dOiZubmxQvXlyaN2+uDL7Wbuu//vpL+vfvLyVKlBBPT08JDQ3VV8mZ1rBhw8TW1lbmz58vP//8sxQoUEDKlSsnMTExIiLyyy+/iKGhofTu3Vs53JedMNyko6FDh4qzs7N4eHhIoUKFpHLlynLx4kUREZk2bZq4ubnJoEGD5Pr163quNHNbvXq1ODg4KIdMZs+eLcbGxsoZKSLJh6gsLCzE399fX2VmehcuXBBHR0dZsGCBctvU1FSGDRumLBMRESG+vr5SsGBBBsV3+OOPP8TExEQmTZok27dvl3bt2olGo1F2ypcuXZLy5ctLtWrVlNctpS0oKEisra3l6NGjsnnzZqlTp440atRI5+wykeTexRcvXijjbuhfly5dknLlysmBAwdEJPkUektLS2UgsTYkzp8/X6pWrZotv0wz3KSTefPmia2trfIN45dffhGNRqN8uxMRmTp1qhQoUEBmzJihrzKzhJEjR0rnzp1FJHkuhty5cys76BcvXsi1a9dEROTq1as8JPUO27dvFzc3NxFJPhOqcOHC0qNHD+V+7aG+p0+fyr179/RSY1bw6tUrady4sXL45M6dO+Ls7KxsS23P7dmzZ8XT05NTPKRBu3ONjo6WNm3a6HwGBgcHS506daRJkyZKwAkMDJSlS5fqo9Qs4dChQ1KgQAERSQ425ubmMn/+fBFJ3saLFy9WDttrZbeAw3CTTgYNGiQ//PCDiIisXbtWLC0tlRdbVFSUstzq1au5Q34L7U5i1KhR8sMPP8iuXbvE3NxcCTaJiYnyyy+/yLRp03R6Gbg9dWk/xHbs2CENGjSQW7duScGCBaVHjx7Ktjp06JAMHTqUoeY9REZGirOzs4SEhEhERIQUKFBAJyQuXrxYOdwcFxenpyozv4MHD4qnp6f4+PjI1atXdQ7nr1u3Try9vaVMmTLSqlUrnV6x7C5lKNFus9u3b4uPj48EBgaKubm5LFy4UFnmxIkT0qJFCzlx4oSyfnYLNiIivPxCOhARhIaGAgBCQkLQvXt3TJw4Eb169UJSUhImTpyIhQsXAgDatWsHQ0PDbD+9vYgoV1TW0l7zpFChQhg9ejQaNGiAOXPmoGfPngCSLxewYsUKhIeHw8TERFkvu19TSt647IT2MhVFihTB3r174ezsjGbNmmHhwoXKtlq3bh3Onj0LMzMzvdSclVhYWKB69erYsGEDypUrh4YNG2Lu3LkAgOfPn2PXrl3YvXs3kpKSkDNnTj1XmzmJCJ48eYKIiAgcPnwYFhYWMDAwUK6m3rJlSwwePBhNmzaFgYEBzp8/D1dXV/0WrWcigsTERJ3LzmhZWloiLi4OI0aMQEBAAHr06AEAePnyJUaPHo3Xr1/Dzc0NQPLnQVqPoXr6TFZZ0dvOKFmxYoWUL19ecubMqTOT5vPnz8XHx0fGjBmTQRVmPTt37pRNmzbpHMIbPHiwGBkZyfbt2+XKlSty6dIlqVevnri5ufGssn9oe6+0Z+acOnVKgoKC5I8//lAOjaxfv17MzMxkxIgRcu3aNblw4YIMGTJErKys5Pz583qrPTNKebZOyjmpREQmTZoklpaWUqdOHZ1ew+HDh0uJEiUkLCwsw+vNamJiYmTz5s3i4OAgderUUdrf7O3i+1tSvZ5mzJghHTt2lCZNmsjq1atFJPmEACcnJ6levbqMHDlS5s6dK7Vr15ayZcsqnwnZ+QxIhpsPkLJr7/Tp0/J///d/8uzZM0lKSpKbN29K3bp1xc3NTXbs2CEiIjdu3BBfX1+pVKkS37D/+O677+T7779Xbvfv319sbW3F3t5eHB0dpWLFivL333/L8+fPxc/PT3Lnzi358uUTd3d3qVGjhvKmze6HolasWCHt2rVT5rFYu3at5M6dW5ydnaVIkSJibW2tzL2yaNEiMTc3l4IFC0rp0qWlXLlyPPskBW33vXZHsGXLFmncuLHUqFFDvvvuO7ly5YqIiPTs2VNKlCghHTp0kLFjx0qHDh3EysqKh0/SoP2svHLlioSEhEhISIhER0eLSPLgbCcnJ2nQoIGyPE+d/9fEiRNFo9HIuXPnRCT5rCgrKyvp1KmTtGzZUgwMDMTf319evHgh9+7dEz8/P3Fzc5O6detKjx49OGfaPxhu3kOfPn10PsCGDBkiNjY2kjdvXrG1tZVBgwbJ/fv35cyZM+Lt7S329vbKNVGqVq3KHfI/njx5Iq1btxZPT0+ZOXOmnDhxQipUqCAnTpyQW7duybFjx6RixYpSpkwZZZzS4cOH5c8//5QTJ05ki1k1/4t2G4wePVoqVaokvXv3losXL4qPj48sWbJEIiIi5PLly9KrVy8xNTWV3bt3i0jygOIDBw5IaGioPHr0SJ9PIVM5cuSIaDQaZYDrzp07xcjISLp27arMiluzZk3ZuXOniCSfFNCiRQupVq2a9OzZUy5cuKDP8jMlbbDZsGGDFCxYUCpVqiT58+eXRo0aybZt20QkeZ6bYsWKSaNGjfRZaqZ06tQpadasmRQoUEBCQkJk0KBBcvjwYeX+TZs2Sd68eWXAgAEiktzz9fLlS52AmJ0/I7UYbv5DTEyMuLi4SPHixeXSpUuybds2KVSokOzcuVPu3LkjU6dOlapVqyrfoh89eiRHjx6VJUuWyL59+5RAwxdbsjt37kjv3r2ldu3a0rZtW2nXrp1Oj9iDBw+kTJky0rBhwzTXz87drCIily9fFpHk7TB58mT58ssvpV27dlKrVi2ds3Rev34t/v7+Ymdnl+Y07JTs5cuXEhgYKDlz5pS5c+fKzz//LJMnT1buv3PnjtSvX19q1qypDBoWSd6+2f21+C6HDx8Wa2tr5VpQ2ssCzJo1S0SSz0DbvHmzWFtbS6tWrfRZaqZ09uxZady4seTNm1ccHR2Vy3doX3Pr1q0TQ0NDZQLOlLLj4OG0MNy8h+fPn0vNmjWVKyaPHj1a5/6VK1dKmTJllDfum7J7j42W9k13+/Zt6dmzpxQsWFA8PDyU+7UBcMGCBVK2bFn2MLzhjz/+EHt7e1m/fr2IJH/QTZgwQVxdXcXS0lIePnwoIv9ux9DQUClUqJAyFwb9a/ny5XL79m0RSd7RTpo0SQwMDMTGxkYJN9r37d27d8Xe3l7Gjh2rt3qzCu17fOLEidKsWTMRSb7ivLOzs/Ts2VNZLiIiQhITE2Xr1q3K1A6k6+bNm9KhQwcxMDBQeg61p3dHR0dL0aJFU139m/7Fs6Xeg6WlJTZt2gR7e3sMHjwYFy9e1DnTp2PHjqhatSrmz5+vc9aKVnY/m0dLo9FARFCoUCGMGjUKjRo1wrVr1zBhwgQAQI4cOQAA+fLlQ2xsLOLi4vRZbqZjbW0NLy8vjBs3Dr/99hsMDAwwbNgwdOrUCblz58Y333yDR48eKdvR3t4eABAVFaXPsjOdFy9eYNiwYWjSpAnu3bsHY2NjfPPNN5gxYwZevHiBmzdvAkh+vSYmJqJgwYKoX78+zpw5o9/Cs5DY2FiUKlUKsbGxqF69OurWrYv58+cDAP744w9s2LABGo0Gvr6+KFasmJ6rzVyWLFmCPn36oEiRIhg8eDB8fX3RsmVLnD17FsbGxgCA+Ph4vH79mvuWd9F3uspKIiIipGHDhmJjYyNHjhzRuW/RokXi6empDJqjt9N+u/v777+lV69e4ubmJsOHD5fnz5/L1atXpW7dulKnTh12+/8j5bez06dPS+fOnaV06dKyYcMGEfn3EJWHh4c0adJEwsLC5NKlSzJy5EixtbVVeijoX3fu3JEvvvhCKlWqJHfv3hWR5G/DU6dOFQMDA5k6darO8g0aNJDOnTuzy/89LVq0SIyNjcXGxkYCAgJ0eq+7du0q/v7+Ehsbq8cKM6eEhAQJCAgQT09Ppe3s2bPy9ddfi7m5uUycOFHmzJkjX3/9tXzxxRcc7vAODDcf6Pnz51KrVi1xcnKSnTt3yt9//y3Pnj2TmjVriq+vLz/83pN2O2nH4JiYmEiBAgWkVatW0qRJE57K+A/tdbS0F10VSb4QZloBZ8qUKeLg4CBWVlby1VdfSdOmTZVj9ZTa3bt3xcXFRSfgvHz5UjlE1blzZxk1apQEBARIrly5eOp8GrQ711u3bsn58+d1gnSXLl3ExMREmQk7MjJShg0bJnZ2dnLp0iW91JuZpfzSZ2FhoTPM4fz589KsWTMxNDSUhg0byvLly3miyn9guPkIkZGRUrt2bcmZM6cUL15cOnToIB4eHsqLjQHn/Wi30/3796Vv375SqFAhmTdvntLObyXJ19d59uyZiIhOUHlbwJk5c6YUK1ZMevfuraxHb3f37l0pWbKkVKxYUSfgTJs2TXLnzi0ODg6yYcMGnhWVwurVq+Xnn39Wbq9du1YKFy4sefPmlfLly0uLFi1EJHmuFm9vbzEyMhI3NzepVq2aFCxYkNMQvIfhw4dLkyZNlKkeRJIvl+Lj46Nzhhk/I9+O4eYjPX/+XJkm/OTJkzxN+SNpg8ytW7dk8uTJynZkQNTttXrw4IHky5dPvv76a6UtrYDz+vVrmTZtGq9vlAbta+ry5cty4sQJOXjwoIgkB5wyZcroBJzo6GgJDAyUPHny8MKNKURERIinp6fUrFlTNmzYIFeuXBFnZ2eZM2eO7N+/XxYvXqxcyVvrl19+kalTp8ovv/zCyQ7T8OOPP0q/fv3k0KFDStuuXbvEwsJC9u/fr7Ps9evX+Rn5nhhuPkFERIT07t1b6RbM7odQRNLeBv/1Jnzz/uy8HbXPPeUsuFevXhURkVWrVomzs7POqbPagFOuXDlZs2ZNxhabhWhfY7///rs4OTlJqVKlxNTUVLp06SL379+XO3fuKAFHe72tly9f6nxzpmRXrlyRBg0aSIMGDWTEiBHSqVMn5TMwKSlJTpw4ofRo07u9evVKFi9eLC4uLlKxYkWpXbu2HDt2TBISEmTo0KHy1VdfyYsXL1Ktl50/I98Xw42kzwuFM2zqbsfQ0FA5evToe3Xnp1wv5UVGs6sbN25Iz5495d69e8r8INevX5fo6GhZs2aNODo66gScY8eOSbNmzaRKlSry4sULfqN7i507d4qVlZUsXLhQ4uLiZNu2baLRaKR169Zy9+5duXPnjri6ukrRokU5N9BbaN+rV65cEW9vb8mfP79UqVIl1XIzZswQd3d3CQ8PV9r4unz7viYiIkL27dsnDRo0kHLlykm1atWkVatW4uHhoTO3Fb2/bB9uUr5gbt68marb9G1vyJTr8VCU7nYaMWKEFC1aVMqUKSMWFhYyYMCAtw4gTLnerFmzpE+fPtn+LIoDBw6ItbW11KpVS4yNjXXOloqNjU0z4Jw8eZI75HeIjIyUHj16yLhx40Qk+b1etGhRadGihVhaWkqjRo0kLCxMwsLCxNPTU2cAN/0r5efetWvXpEGDBpI3b15ZvHixznJbt24VBwcHbse3mD17tnTv3l0CAgKUAddae/fulQkTJkiePHlEo9FI9+7d9VRl1pbtw43WsGHDpEyZMmJubi7ffPONcr0ZkdQBJ+XtpUuXSr9+/Thi/R/Tpk2TfPnyKdOFf/vtt2JqairHjx9PtWzK7bhw4UIxMzOTtWvXZlitmZF2mwQGBopGoxFPT89Uk5zFxsbK2rVrxdnZWby9vfVRZpYTFxcn69atk+vXr8vTp0+lQoUK0q1bNxERCQoKEo1GIz4+PnLv3j1+WUmD9nUZExMjIqJ8Abl69ar4+PhIrVq1ZMGCBSKSPF7p22+/lVKlSvGw3j9ShsIRI0aIjY2NNG7cWCpXriw2NjYSEhKSap2bN2/Kjz/+KB4eHhzQ/hGybbhJ+WILCgqSwoULS3BwsMyfP1+KFi0qzZs31xnMpX1zp9whL1iwQMzNzWXjxo0ZV3gmcv/+feX/2u3Zpk0bmT59uogkX5HayspKmYI95aG7lNt/wYIFYmFhoQyKza7eDM2jR48WZ2dnad++faozTKKjo2XlypXi4uKijBGhd9OOY1q1apV4enoqg4fXrFkjtWrVksKFC3NOoDRoX5fbtm2TJk2aSO3ataVly5bKa/LKlSvi4+MjlpaW4u7uLm3atJHSpUtzGoJ/pHxf37t3T4YNG6Z82bt+/bq0b99ejIyMlICTkJCgfD5eu3ZN8ufPL0FBQRlfeBaXbcON1sGDB2XIkCGydOlSpS0kJEQqVKggzZo105m6PmXvzIIFC8TS0lKZCj+76dSpk1SsWFEZ7CqSvMMtW7as7NmzR44cOSLm5uYyf/58EUn+5jx+/HidC8CJJE/2ZWFhkW23o5b2A3D//v3y008/KTvivXv3ipOTk7Rr106n+1q74+AYpQ/3/fffyxdffCEREREiktxrO3v2bI6be4dNmzaJkZGRDBs2TPr37y++vr5ibGwsv//+u4gk9+A0btxY8uTJIxMnTpQHDx7ot+BMYNGiRTq3g4ODxdDQUMqWLSvXr19X2u/evSsdOnQQExMTZXLY169fK58JderUke+//z7jCleJbBduUvYYXL16VczMzMTAwEAmTJigs9yRI0fEzc1NWrRoIbt27dK5b+HChdl+h3zlyhWxsbERHx8fnYATEBAgxYsXF2NjY525MJ48eSK1atWSmTNnKm0zZswQExMT+e233zK09sxG+yG2fv16sba2lqFDh+p86929e7cUKVJE2rdvL1u2bJFx48aJRqPhtbc+UmhoqBgbG0u1atXkq6++EgsLCzl79qy+y8q04uPjxcfHR4YNG6a0xcbGysCBA8XY2FjOnTsnIsmn2Ddv3pxjvyT5OnAVKlTQCSkhISHSokULMTExkdOnT4vIv+/9u3fvSqdOnUSj0ehMFvnrr79K7ty55eLFixn+HLK6bBdutPr16yebNm2Sbdu2iaOjo/j4+KT6gDty5IgUKFBARowYobQtXbpUcubMma0PoWjHJNy8eVPy5Mkj9evXV0b0Hzx4UKpWrSru7u7y9OlTERF5/Pix+Pj4SNWqVXV6v2bOnJltu1vf7CU4fPiwWFpapvq2FxcXJyLJg4xdXV2lXLly4uTkJCdPnsywWtUoJCREOnToIH379pW//vpL3+VkWhs3bpQpU6ZIqVKllF7YpKQkSUxMlOjoaKlbt650795duaAje7+SvXr1Svki/eeffyrtp06dkrp164q9vb3ymakNOGFhYTJu3DidMV8PHz6UGzduZGDl6pFtwk3K455Hjx6VPHnyKGNqtAGne/fuqT7o/vrrL2WH/OzZMxkyZIhs2rQp4wrPZLRv2JTHhPPkySP16tWTW7duiYjIsmXLxNPTU2xsbMTT01Pc3NzE3d1d+eDL7gM2f/zxR1mzZo2yk9C2aWceffbsmWzZskXatm0rHh4esmXLFhFJPkX87NmzOmOd6OMlJiby9OR3OHnypOTJk0fWrVsnjRs3lq+//loZUKzdbh06dJAmTZros8xMJ+Vr6tixY6LRaHR6vU6dOiW+vr7i6OiYKuBopRx3Qx9HI5LGZaxVbM6cOXj8+DFy5MiBUaNGQUSg0WiwZcsW9OnTB97e3hg4cCDKlCmjs15iYiIMDQ0RHR0Nc3NzPVWfeYSGhqJAgQKws7PDjRs3ULlyZbi7u2Pp0qVwdHTE1atXsWfPHkRGRsLR0RFt27aFoaEhXr9+rVy1Orvq2bMn+vfvjzJlyiivq0WLFmHYsGGYPHkyfv/9d2g0GhgbGyNXrlxYt24dbt++DTs7O32XTtnE9evXsWrVKsTFxWHixIlYuHAhli9fjgYNGmDIkCEwMTEBAPj5+UGj0WDhwoXIkSMHNBqNnivXr3379uH+/fto3749evfujYSEBFSqVAkDBw5EQEAAJkyYAAA4deoUxowZgwsXLmDLli2p9jeUDvQcrjLUw4cPxcvLSzQajfTq1UtEkrtRtan5jz/+ECcnJ2nevDnnZ3hDym8RW7duFXt7e5k9e7Zyquf169clT548Urdu3bduu+x+uvyKFStk1apVyu19+/bJzz//LC9fvpRbt25J3759JX/+/NK1a1dlIPu1a9fEzc1NZwAi0ecUGRkpFStWFFtbWxk4cKCIJPckDB48WCpVqiR16tSRH3/8UTp37izm5uY8rCfJPS9RUVFSt25dqVmzpjRs2FAsLS2VU7iXLFkiOXLk0BnicOrUKfHw8JCmTZvqq2xVU3W4SavL+cSJE9KsWTOdN2XKgPPrr79K06ZN2SWYQsrtuHjxYpkyZYoYGRmJg4ODzJs3Twk4165dk7x584qvry/nZXhDdHS0eHl5SZUqVZQz8zp06CAODg4SFBSkBL83zzIZOnSolC9fXhm/RJQRQkNDpXjx4uLq6qoMbn/9+rX8/PPP0q5dO6lcubI0b95cGUxMyZ4+fSolS5YUjUYjgYGBSvvLly+VgDNy5Eil/fLly9zXfCaqDTcpXzCPHz/W+eZ77do1qVu3rjg4OCg74ZQBJ63HIJHRo0eLlZWVrFmzRoKCgqRp06Zib2+vE3CuX78uGo1GAgIC9Fxt5nP//n1p2bKl1KhRQ9atWyciIl26dJESJUrIypUrlfEMIiKHDh2SXr16ibW1tXJmBVFGOnv2rJQrV066d++e6mSL2NhYDh5Ow7Nnz8TX11dq1KghdevW1empjY2NlaVLl4qJiYn069dPZz3ua9KfKsNNypAyevRoqVq1quTOnVuaNWsmM2bMEBGR8+fPS8OGDcXR0VG5NEB2P2zyNklJSfLo0SMpXbq0Mguplp+fn1hbW8u8efPk8ePHIiKc5fUNSUlJyo7gwoUL4uPjIx4eHsrkjx07dpSSJUvKypUrJTY2Vv7++28ZPXq0eHt765wWSpTRQkNDxc3NLc2TLejtHjx4IL6+vlK7dm355ZdflPb4+HiZPHmy1KpVi4PZPzNVhhut77//XmxtbWXTpk1y8+ZN+fLLL6Vo0aJKmDlz5ow0bNhQcuTIkeqaUqQrIiJCXFxcZMmSJSKie9VqDw8PKV68uCxYsECeP3+utDPgJNN+iAUHB0urVq3E09NTzMzMxMnJSZnjp2PHjuLi4iJBQUGSmJgoT58+lWfPnumxaqJkoaGhUrlyZWnTps1brxFHqd28eVMaNGggdevWlWXLlsnr16/lq6++km+//TbNGe8pfakm3Ghnak1KSpKkpCS5f/++eHp6Kt+O9+3bJ2ZmZsrOWfuiCg0NlSFDhrDXJoW3veHq1Kkj1atXV25r52Dp0KGDlCpVSpycnJSBsHzT6jp69KiYmZnJ0qVL5fLly3Lt2jWpVauWVKpUSZnltUuXLmJjY6McsiLKLI4fPy41a9bkNAQf6ObNm9KsWTMpVaqUFClSRL744gvlc5OfkZ+XKsLN4MGDZcCAAcphEZHkgV3u7u4SEREhGzdu1LkUQGxsrKxYsSLVBQkZcHSP/d65c0fu378v4eHhIpJ8KM/Ozk4Z3a99c7Zt21ZCQ0Oldu3aUqNGjYwvOgtYuHChlC5dWueK5/fu3ZPq1atL4cKFZfPmzSIi0qtXL54ZRZlSyt5aen/379+XP/74Q5YsWaL0ZrNX+/NTxYQjsbGxOHbsGGbPno1vvvkGNjY2EBFERkaif//+2Lp1KyZPnoxevXoBAMLCwrB69WrY2tqiWLFiyuMYGhrq6ylkCiICAwMDAMDo0aOxfft2hIWFoXTp0mjWrBkGDBiA5cuXo1u3bihVqhRKlSqF27dvIyoqChUqVECdOnWwbds2PT+LzMnU1BSJiYmIjo6GqakpEhISUKBAAcybNw9Vq1bFd999h8TERMyfP1/fpRKlSTu3DX2Y/Pnz4+uvv1ZuJyYmZvu5vjKCgb4L+BTyz/yDc+fORf369bFr1y7MnDkTjx49Qt68eTF69Gj8/vvvqF+/Pnr37o2kpCTExsZiyJAhEBF4e3vr+RlkLtoJuMaPH4+5c+di5MiRmD59OqpXr46hQ4di/Pjx8PHxwenTp1G/fn3Y29ujVq1auHjxIgDg8uXLKFiwIBISEpS/DSXz9PTE7du3MXv2bABAzpw5AQDx8fFwd3eHq6sr3Nzc9FkiEWWA7P4lOqNk6fgo/8wuDABt27bF+fPnsW7dOhgaGqJ///5o06YNrly5gh9//FGZCfbBgwd4+vQpTp06BUNDQyQlJSm9FQRERUVh//79CAwMRJMmTQAAMTExKFKkCAYNGgRnZ2e0a9cOP/30k7LO06dPMX78eOzYsQMHDx5Udtz0r2LFimHx4sXo2rUrEhMT4e/vDysrK2zatAlOTk6YNWsWLCws9F0mEZEqZOlwow0lAwYMwMmTJ5E3b14YGhpi5syZEBF8++23GD9+PDw8PLBy5UpYWFjAy8sL3333HXLkyMFLAaQhKSkJf/31F2rXrq205cqVCy1btsS2bdtw7NgxtGvXTgmFt2/fxq+//oq9e/diz549KF26tB6rz9zat28PQ0ND9OjRA2vWrIGBgQGePXuG3bt3M9gQEaWjLH9tqU2bNqFbt27Ys2cPSpUqBWNjY/Tv3x979+5Fy5Yt0b9/f+TJkydVkNH25GRnKXu+Urb17NkTjx8/xqRJk1CiRAnlvu7du+PJkyfYuHGjzjp37tyBqakpbG1tM6LsLC8sLAznzp3Dy5cv4eHhAScnJ32XRESkKln+eExUVBSsrKxQoEABGBsbAwBmzZoFT09PTJs2DbNnz8bDhw+VYKPNctk92CQlJSnBJjw8HHfv3gWQPO6mYcOGuHTpEpYsWYLLly8DAF68eIEbN27A2dk51WMVKlSIweYDODk5oVGjRmjdujWDDRHRZ5Blj8loD4vkzJkTCQkJiI2NBZA8QNPIyAhjx47Fxo0bsWzZMuTPnx89evQAgGx/1VpA96yoMWPGYOPGjXj06BHy5cuHoUOHon379oiKisLEiROxY8cO2NvbIzIyErGxsZg8ebLyGNyWRESUGWWZw1JvG/j7+vVruLi4oGjRotiyZYsymPXcuXOYOHEi3NzcMGjQoGzfU5OWH3/8EdOnT8fMmTNhZ2eHZcuW4ezZs+jSpQuGDBmCI0eO4Ny5czh16hScnZ0xePBgjlUiIqJML0uEm5TBZtWqVTh79ixMTU1Rvnx5tGjRAmfOnEGjRo1QqFAhfPvtt7C0tMSUKVOQN29e/PLLLwA4xialpKQkPH/+HA0aNEDHjh3Rp08f5b6hQ4di/fr1WLVqFapVq5ZqXW5HIiLK7LLEmBttsBkyZAgGDx6Mq1ev4tChQ2jVqhWGDRsGV1dXHDhwACKCoUOHomvXroiOjsby5csBJB9Cye475JQZ1sDAABYWFoiMjFS2bVxcHABg8uTJsLOzw6xZs1KtB3CsEhERZX5Z5tjCn3/+iVWrVuH3339H1apV8erVK2zatAmdO3eGsbExxo0bh8OHDyMsLAwJCQkoWrQoDAwMeAgFuuNj1q5di6dPn6Jv375wdnZGUFAQevXqBWNjY2W8UoUKFRAVFQWAY5SIiCjryTJ7fe2swxUrVgSQPBV469atERUVhe+++w6NGjWCu7u7ztknSUlJ2T7YpDykd+HCBUyePBkiAgcHB/zwww9o2rQpWrdujeDgYKVX5uzZs8p2JiIiymoy5WGppKSkVG02Nja4desWzpw5A+DfwyUeHh4wMjJCdHR0qnU487DuIb0xY8bA1NQUt2/fxogRI3D8+HH89NNPOH78OFxcXNCwYUN4enri2bNnmDZtmp4rJyIi+jiZbu+fsqdhx44dCA4OxsWLF1GhQgXUqlULs2bNwpkzZ5TDJba2tsibNy/i4+P1WXamtmLFCixZsgQjRozAli1bcPHiRTg6OiIoKAhRUVE4dOgQWrZsiRIlSqBevXo4d+6cclYUERFRVpNpz5YaPnw4Zs+eDQcHB4SFhWHRokV49eoV1q1bByMjI7Rv3x758+fHlClTEBERgaNHj3Kw61v873//w4EDB3DgwAEAyb059+7dQ/PmzfH06VNMmjQJzZs3B/Dv+ByeFUVERFlVpum50WYsEUFYWBgOHTqE3bt34+jRo5gwYQL8/f2RlJSETp06oWDBgvD398fQoUORlJSEkJAQGBoaIjExUc/PInPRblNjY2O8evUK8fHxMDAwQEJCAgoWLIiJEyfiwYMHmDt3LtauXQvg3wHEDDZERJRVZYpwk/JSAM+ePUNCQgKqV6+OypUrI0+ePBgyZAgmT56M/v374/nz55g5cyZu376N7du3Y9euXciZMydev37NHfIbtNu0SZMmOH36NCZNmgQAykSH8fHx8PHxgUajwdKlS3loj4iIVCFTnEqkHWMzcuRI7N69G1evXkXhwoXRpUsXlCxZEgAwaNAgaDQaDBkyBOHh4Rg1ahTMzMwA8Kyo/1K2bFksWbIEPXr0QExMDFq3bg1ra2vMnj0bVatWRdOmTVGmTBkcPHgQXl5e+i6XiIjok+g1EaQcPLx27VosX74cw4cPx82bN7Fo0SIsWbIE/fr1Q+HChQEAAwcORGxsLLZu3QpTU1PlcXhW1H/r0qULcufOjT59+mDNmjUQEeTLlw8BAQEIDw9HsWLFkC9fPn2XSURE9MkyxYDiAwcOYN26dfDw8ECnTp0AAPPmzUNgYCDat2+P3r17KwEH+HfQKy/e+OH+/vtv3L17FwkJCahWrRoMDAwwfPhwbNy4Efv27YO9vb2+SyQiIvokej+W8/DhQ3Tr1g3h4eEoUaKE0t6nTx+ICCZOnAhDQ0N069YNzs7OAMBg8wkKFCiAAgUKAEie1G/SpEnYtm0b9uzZw2BDRESqoPfjOfb29vjtt9/g4OCArVu34vz588p9ffv2xYgRIzBp0iTs2rVLZz0Gm0/z+vVrxMfHI1++fDhw4ABcXV31XRIREVG6yBSHpYDkKf/9/PxQsWJFDBgwAGXKlFHu++2339C4cWOeDfUZJCQkKGdPERERqUGmCTcAcPr0aXTv3h3u7u4YOHAgSpcurXM/J5YjIiKi/5Kpwg2QHHB69uyJwoULY/LkyShSpIi+SyIiIqIsRO9jbt5UoUIFzJkzB7lz59Y5Q4qIiIjofWS6nhst7dlQKefCISIiIvovmTbcAODp3kRERPTBMnWXCIMNERERfahMHW6IiIiIPhTDDREREakKww0RERGpCsMNERERqQrDDRGp3v79+6HRaPD8+fP3XsfJyQkzZsz4bDUR0efDcENEetelSxdoNBr06tUr1X19+/aFRqNBly5dMr4wIsqSGG6IKFNwdHTE2rVr8fLlS6Xt1atXCAoKQqFChfRYGRFlNQw3RJQpuLm5wdHREb/99pvS9ttvv6FQoUKoUKGC0hYXF4f+/fsjX758MDExQfXq1XHixAmdx9q2bRtKlCgBU1NT1K5dG2FhYal+36FDh/Dll1/C1NQUjo6O6N+/P2JiYj7b8yOijMNwQ0SZRteuXbF8+XLl9rJly+Dn56ezzNChQ7Fhwwb8/PPPCA0NRbFixeDt7Y2IiAgAwN27d9GsWTM0bNgQZ86cQffu3TFs2DCdx7hx4wbq16+P5s2b49y5cwgODsahQ4fQr1+/z/8kieizY7ghokyjQ4cOOHToEG7fvo3bt2/j8OHD6NChg3J/TEwM5s+fjylTpsDHxwelS5fG4sWLYWpqiqVLlwIA5s+fj6JFi2LatGkoWbIk2rdvn2q8TmBgINq3b4+BAweiePHiqFq1KmbNmoWVK1fi1atXGfmUiegzyKHvAoiItGxtbdGgQQOsWLECIoIGDRrAxsZGuf/GjRtISEhAtWrVlLacOXOicuXKuHTpEgDg0qVL8PDw0HlcT09Pndtnz57FuXPnsHr1aqVNRJCUlIRbt26hVKlSn+PpEVEGYbghokyla9euyuGhuXPnfpbfER0djZ49e6J///6p7uPgZaKsj+GGiDKV+vXrIz4+HhqNBt7e3jr3FS1aFEZGRjh8+DAKFy4MAEhISMCJEycwcOBAAECpUqWwefNmnfWOHj2qc9vNzQ0XL15EsWLFPt8TISK94ZgbIspUDA0NcenSJVy8eBGGhoY69+XKlQu9e/fGkCFDsGPHDly8eBH+/v6IjY1Ft27dAAC9evXCtWvXMGTIEFy5cgVBQUFYsWKFzuN89913CAkJQb9+/XDmzBlcu3YNmzZt4oBiIpVguCGiTMfCwgIWFhZp3jdx4kQ0b94cHTt2hJubG65fv46dO3fC2toaQPJhpQ0bNmDjxo0oX748FixYgB9//FHnMcqVK4cDBw7g6tWr+PLLL1GhQgWMHj0aDg4On/25EdHnpxER0XcRREREROmFPTdERESkKgw3REREpCoMN0RERKQqDDdERESkKgw3REREpCoMN0RERKQqDDdERESkKgw3REREpCoMN0RERKQqDDdERESkKgw3REREpCoMN0RERKQq/w/tXGBdBG2NPwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "scores = dict(sorted(scores.items(), key=lambda item: item[1], reverse=True))\n",
    "df = pd.DataFrame(list(scores.items()), columns=[\"Model\", \"Score\"])\n",
    "\n",
    "palette = sns.color_palette(\"muted\", len(df))\n",
    "sns.barplot(x=\"Model\", y=\"Score\", data=df, palette=palette)\n",
    "\n",
    "plt.ylim(0, 0.5)\n",
    "\n",
    "plt.xticks(rotation=45, ha=\"right\")\n",
    "plt.title(\"Model Comparison by Score\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FProvIIoKb91"
   },
   "source": [
    "### This graph clearly shows that ModernBert clearly outperforms its respective Bert series models. But this retrival comparison is with OpenAI embedding and on Uber-10K dataset.\n",
    "\n",
    "Note: Results may vary on other datasets"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
